Nov 01 05:44:50.195218 ip-10-0-138-190 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 05:44:50.547486 ip-10-0-138-190 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:44:50.547486 ip-10-0-138-190 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 05:44:50.547486 ip-10-0-138-190 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:44:50.548105 ip-10-0-138-190 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:44:50.551109 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551029    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:50.551109 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551110    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551114    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551118    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551121    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551124    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551127    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551130    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551133    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551136    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551139    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551142    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551145    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551147    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551150    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551152    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551155    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551158    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551160    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551163    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551165    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:50.551173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551168    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551171    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551174    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551177    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551181    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551183    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551187    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551189    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551192    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551194    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551197    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551199    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551202    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551204    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551207    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551209    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551212    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551214    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551217    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:50.551656 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551219    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551222    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551224    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551226    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551229    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551231    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551234    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551236    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551238    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551241    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551244    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551247    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551249    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551251    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551255    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551258    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551261    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551263    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551266    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551268    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:50.552095 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551271    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551273    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551276    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551278    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551281    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551283    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551285    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551288    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551291    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551293    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551295    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551298    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551300    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551303    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551305    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551308    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551310    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551313    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551315    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551319    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:50.552576 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551321    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551325    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551327    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551330    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551334    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551338    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551342    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551345    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551348    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551350    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551353    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551356    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551358    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551361    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551363    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551366    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551368    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551372    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:50.553051 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551376    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551379    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551382    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551384    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551387    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551389    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551393    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.551395    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552201    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552208    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552211    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552214    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552216    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552220    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552223    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552225    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552229    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552232    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552235    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:50.553496 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552237    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552240    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552243    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552245    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552248    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552250    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552253    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552255    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552257    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552260    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552263    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552265    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552268    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552270    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552273    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552275    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552278    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552280    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552283    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552285    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:50.553949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552288    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552290    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552292    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552295    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552297    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552300    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552302    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552305    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552308    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552310    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552314    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552318    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552322    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552326    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552329    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552333    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552335    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552338    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552340    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:50.554416 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552343    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552345    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552348    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552350    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552353    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552355    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552358    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552360    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552362    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552365    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552367    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552370    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552372    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552375    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552377    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552380    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552382    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552386    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552388    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552391    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:50.554868 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552393    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552396    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552399    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552401    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552404    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552407    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552413    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552416    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552418    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552421    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552423    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552426    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552428    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552430    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552433    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552435    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552438    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552440    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552443    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552461    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:50.555349 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552464    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552466    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552469    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552472    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552474    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552477    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552480    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552482    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552485    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552487    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552490    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552494    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552496    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552499    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552504    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.552508    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553014    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553019    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553024    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:50.555832 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553027    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553030    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553033    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553036    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553039    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553041    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553044    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553046    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553049    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553051    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553055    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553059    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553063    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553067    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553070    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553073    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553075    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553078    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553080    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:50.556305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553083    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553085    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553087    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553090    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553092    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553096    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553098    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553100    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553103    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553105    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553108    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553111    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553115    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553117    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553120    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553122    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553125    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553127    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553130    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:50.556763 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553133    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553135    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553138    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553140    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553143    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553145    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553147    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553150    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553152    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553155    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553157    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553160    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553163    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553165    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553168    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553179    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553182    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553185    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553187    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:50.557212 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553190    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553192    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553195    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553197    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553200    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553203    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553210    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553213    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553216    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553218    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553221    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553223    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553226    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553228    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553231    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553233    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553236    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553238    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553241    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553243    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:50.557717 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553246    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553248    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553251    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553253    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553256    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553259    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553261    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553264    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553266    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553269    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553271    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553274    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553276    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553279    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553282    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553285    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553287    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553289    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553293    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553299    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:50.558184 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553302    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553305    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553307    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553310    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553312    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.553314    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554160    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554170    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554176    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554181    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554186    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554189    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554194    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554198    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554201    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554204    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554207    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554210    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554213    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554216    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554219    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554221    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554224    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 05:44:50.558667 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554227    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554231    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554234    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554237    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554240    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554244    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554247    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554251    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554254    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554264    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554267    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554270    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554273    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554276    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554279    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554283    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554286    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554288    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554291    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554294    2575 flags.go:64] FLAG: --enable-server="true"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554299    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554304    2575 flags.go:64] FLAG: --event-burst="100"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554307    2575 flags.go:64] FLAG: --event-qps="50"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554310    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 05:44:50.559235 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554312    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554315    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554320    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554322    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554326    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554328    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554331    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554334    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554337    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554340    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554343    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554345    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554348    2575 flags.go:64] FLAG: --feature-gates=""
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554352    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554356    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554359    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554362    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554365    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554374    2575 flags.go:64] FLAG: --help="false"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554377    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554381    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554384    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554386    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 05:44:50.559831 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554390    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554393    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554396    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554399    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554401    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554405    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554408    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554411    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554414    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554417    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554420    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554424    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554427    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554429    2575 flags.go:64] FLAG: --lock-file=""
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554432    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554435    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554438    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554443    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554457    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554460    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554463    2575 flags.go:64] FLAG: --logging-format="text"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554466    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554469    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554472    2575 flags.go:64] FLAG: --manifest-url=""
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554475    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 05:44:50.560419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554480    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554483    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554487    2575 flags.go:64] FLAG: --max-pods="110"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554490    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554493    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554497    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554500    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554503    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554505    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554508    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554515    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554518    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554522    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554525    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554528    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554531    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554534    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554537    2575 flags.go:64] FLAG: --port="10250"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554540    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554542    2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-04599b72d757a00b1"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554546    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554548    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554551    2575 flags.go:64] FLAG: --register-node="true"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554554    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554558    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554561    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 01 05:44:50.561070 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554563    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554566    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554570    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554573    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554576    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554579    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554581    2575 flags.go:64] FLAG: --runonce="false"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554584    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554587    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554590    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554593    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554596    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554599    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554602    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554605    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554608    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554611    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554613    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554617    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554620    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554623    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554626    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554631    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554634    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554637    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 05:44:50.561710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554644    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554647    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554650    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554653    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554656    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554659    2575 flags.go:64] FLAG: --v="2"
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554663    2575 flags.go:64] FLAG: --version="false"
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554667    2575 flags.go:64] FLAG: --vmodule=""
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554671    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.554674    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554867    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554873    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554876    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554879    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554882    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554885    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554887    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554890    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554896    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554899    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554901    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:50.562287 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554904    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554906    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554909    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554911    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554915    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554918    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554920    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554933    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554936    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554938    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554947    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554950    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554953    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554955    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554958    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554960    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554963    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554965    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554968    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:50.562885 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554970    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554973    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554975    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554979    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554982    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554984    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554987    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554990    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554992    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554994    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.554997    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555005    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555008    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555011    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555013    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555016    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555019    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555022    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555025    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555027    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:50.563342 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555030    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555032    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555035    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555037    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555040    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555042    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555045    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555047    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555050    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555052    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555054    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555057    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555059    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555062    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555064    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555068    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555071    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555073    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555076    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555078    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:50.563838 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555080    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555083    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555085    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555089    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555099    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555102    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555105    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555107    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555110    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555113    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555116    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555118    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555121    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555123    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555126    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555129    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555131    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555134    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555136    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:50.564305 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555139    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555143    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555147    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555150    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555153    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555156    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555159    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555161    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555165    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555168    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555170    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555173    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555175    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555177    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555180    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555183    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:50.565115 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555185    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.555212    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555272    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555277    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555280    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555283    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555286    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555290    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555293    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555295    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555298    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555300    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555303    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555305    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555308    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555310    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:50.565519 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555313    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555316    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555318    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555320    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555323    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555325    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555328    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555330    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555333    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555336    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555339    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555341    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555344    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555346    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555348    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555351    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555354    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555357    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555359    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555368    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:50.565897 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555371    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555373    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555376    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555379    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555382    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555384    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555387    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555389    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555392    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555394    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555397    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555399    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555402    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555404    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555407    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555409    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555412    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555414    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555417    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555419    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:50.566410 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555423    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555426    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555428    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555433    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555436    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555439    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555443    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555457    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555460    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555462    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555465    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555467    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555476    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555478    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555481    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555485    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555488    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555490    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555492    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:50.566904 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555495    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555498    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555503    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555506    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555509    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555511    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555514    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555517    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555519    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555522    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555524    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555527    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555529    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555532    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555535    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555537    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555540    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555542    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555545    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:50.567360 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555547    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555551    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555553    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555556    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555559    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555562    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555564    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555567    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555570    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555574    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555576    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555579    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555581    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.555584    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.562085    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.562100    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.562117    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.562121    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562176    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:50.568107 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562182    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562185    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562189    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562191    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562196    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562200    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562203    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562206    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562208    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562211    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562213    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562216    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562219    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562222    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562224    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562226    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562229    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562232    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562234    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:50.568772 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562237    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562239    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562242    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562245    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562247    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562250    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562253    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562255    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562258    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562262    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562265    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562267    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562270    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562272    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562275    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562277    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562280    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562282    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562285    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:50.569279 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562287    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562290    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562292    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562296    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562300    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562303    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562306    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562308    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562311    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562314    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562316    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562319    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562322    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562324    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562327    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562329    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562332    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562335    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562337    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:50.569768 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562340    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562342    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562345    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562348    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562355    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562358    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562361    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562363    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562366    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562368    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562371    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562373    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562376    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562378    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562381    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562383    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562386    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562388    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562390    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562393    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:50.570234 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562395    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562398    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562400    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562403    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562405    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562408    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562410    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562413    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562415    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562418    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562420    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562423    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562425    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562428    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562430    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562433    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562436    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562439    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562441    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562444    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:50.570746 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562460    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562463    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562465    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562467    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562470    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562472    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562475    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562478    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.562505    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562640    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562644    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562646    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562649    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562651    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562654    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562656    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:50.571253 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562659    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562662    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562664    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562667    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562669    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562671    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562674    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562676    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562678    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562681    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562684    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562687    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562689    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562691    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562694    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562696    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562699    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562701    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562703    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562706    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:50.571760 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562708    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562710    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562713    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562715    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562718    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562720    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562722    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562725    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562727    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562729    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562732    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562734    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562736    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562739    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562742    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562744    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562746    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562749    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562751    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562753    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:50.572207 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562756    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562758    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562761    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562763    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562766    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562768    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562771    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562773    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562776    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562778    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562781    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562783    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562785    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562788    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562790    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562792    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562795    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562797    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562799    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:50.572758 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562801    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562804    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562806    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562809    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562811    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562814    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562816    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562819    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562821    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562824    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562828    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562831    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562834    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562836    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562839    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562841    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562844    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562846    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562848    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:50.573218 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562851    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562853    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562856    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562858    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562861    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562863    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562865    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562868    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562870    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562874    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562877    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562880    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562883    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562886    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562888    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562891    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562893    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562896    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562898    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562901    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:50.573709 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562904    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.562926    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562955    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562959    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562961    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562964    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562966    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562968    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562971    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562974    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562976    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562978    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562981    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562983    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562985    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562988    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:50.574173 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562991    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562993    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562995    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.562998    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563000    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563002    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563005    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563007    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563010    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563012    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563014    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563017    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563019    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563022    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563024    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563026    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563029    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563032    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563034    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:50.574691 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563037    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563039    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563042    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563044    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563046    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563049    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563052    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563054    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563056    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563059    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563061    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563063    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563066    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563070    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563072    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563074    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563077    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563079    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563081    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:50.575125 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563084    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563086    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563088    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563091    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563093    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563095    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563098    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563100    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563103    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563105    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563108    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563110    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563112    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563115    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563117    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563121    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563124    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563127    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563134    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:50.575588 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563136    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563139    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563141    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563144    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563146    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563148    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563151    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563153    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563156    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563158    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563161    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563163    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563166    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563168    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563172    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563175    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563177    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563180    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563182    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:50.576055 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563185    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563187    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563190    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563192    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563194    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563197    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563199    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563201    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563204    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563207    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563209    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563212    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563215    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563218    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563220    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:50.563222    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.563279    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.565888    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.566806    2575 server.go:1014] "Starting client certificate rotation"
Nov 01 05:44:50.576559 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.566905    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:44:50.576995 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.567382    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:44:50.576995 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.571355    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:44:50.576995 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.574395    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:44:50.584527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.584512    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 01 05:44:50.587242 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.587228    2575 log.go:25] "Validated CRI v1 image API"
Nov 01 05:44:50.588110 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.588086    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 05:44:50.590821 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.590798    2575 fs.go:135] Filesystem UUIDs: map[79bc8aad-df98-42fa-845b-5444c4e26efa:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9a8b236c-6760-4f6b-a627-33f1c418f808:/dev/nvme0n1p4]
Nov 01 05:44:50.590895 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.590818    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 05:44:50.595855 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.595737    2575 manager.go:217] Machine: {Timestamp:2026-11-01 05:44:50.594434706 +0000 UTC m=+0.291593454 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100236 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f2d0739f46b3983ffebe90f2e274c SystemUUID:ec2f2d07-39f4-6b39-83ff-ebe90f2e274c BootID:eba5dcef-5e2f-44e6-bb67-0f61318124ba Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:22:74:26:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:22:74:26:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:52:35:90:02:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 05:44:50.595855 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.595849    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 05:44:50.596016 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.595947    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 05:44:50.598384 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.598357    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 05:44:50.598558 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.598386    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-190.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 05:44:50.598653 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.598562    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 05:44:50.598653 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.598575    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 05:44:50.598653 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.598612    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:44:50.598653 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.598630    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:44:50.599411 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.599398    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 05:44:50.603105 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.603090    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 05:44:50.603241 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.603230    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 05:44:50.603767 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.603756    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 05:44:50.603821 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.603773    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 05:44:50.603821 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.603787    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 05:44:50.603821 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.603802    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 01 05:44:50.603821 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.603813    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 05:44:50.604172 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.604157    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:44:50.607152 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.607136    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 05:44:50.610965 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.610952    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:44:50.611682 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.611671    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 05:44:50.611715 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.611701    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612727    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612781    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612793    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612805    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612815    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612825    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612840    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612852    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612865    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612882    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612911    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 05:44:50.613355 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.612926    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 05:44:50.613785 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.613416    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 05:44:50.613785 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.613462    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 05:44:50.617822 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.617808    2575 server.go:1263] "Started kubelet"
Nov 01 05:44:50.617932 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.617905    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 05:44:50.618006 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.617958    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 05:44:50.618042 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.618022    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 05:44:50.618892 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.618876    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 05:44:50.619176 ip-10-0-138-190 systemd[1]: Started Kubernetes Kubelet.
Nov 01 05:44:50.621616 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.621600    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 05:44:50.623533 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.623507    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 05:44:50.623697 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.623677    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 05:44:50.623697 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.623694    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 05:44:50.623861 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.623744    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-190.us-west-1.compute.internal\" not found"
Nov 01 05:44:50.624500 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624469    2575 factory.go:55] Registering systemd factory
Nov 01 05:44:50.624500 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624481    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 05:44:50.624500 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624496    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 05:44:50.624701 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624486    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 01 05:44:50.624701 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624618    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 05:44:50.624785 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624731    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 05:44:50.624785 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624740    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 05:44:50.624923 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624911    2575 factory.go:153] Registering CRI-O factory
Nov 01 05:44:50.625003 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624927    2575 factory.go:223] Registration of the crio container factory successfully
Nov 01 05:44:50.625003 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.624975    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 05:44:50.625099 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.625007    2575 factory.go:103] Registering Raw factory
Nov 01 05:44:50.625099 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.625022    2575 manager.go:1199] Started watching for new ooms in manager
Nov 01 05:44:50.625891 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.625876    2575 manager.go:319] Starting recovery of all containers
Nov 01 05:44:50.638249 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.638230    2575 manager.go:324] Recovery completed
Nov 01 05:44:50.644393 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.644380    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:50.646008 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.645990    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:50.646069 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.646025    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:50.646069 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.646038    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:50.646630 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.646546    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 01 05:44:50.646630 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.646554    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 05:44:50.646630 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.646559    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:50.646630 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.646570    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 05:44:50.647323 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.647309    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:50.647389 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.647341    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:50.647389 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.647356    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:50.648875 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.648863    2575 policy_none.go:50] "Start"
Nov 01 05:44:50.648922 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.648879    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 05:44:50.648922 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.648889    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 05:44:50.650867 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.650857    2575 policy_none.go:44] "Start"
Nov 01 05:44:50.689122 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.688967    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 05:44:50.692156 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.692142    2575 manager.go:341] "Starting Device Plugin manager"
Nov 01 05:44:50.692205 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.692180    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 05:44:50.692205 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.692191    2575 server.go:87] "Starting device plugin registration server"
Nov 01 05:44:50.692464 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.692439    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 05:44:50.692510 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.692480    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 05:44:50.692614 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.692596    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 05:44:50.692708 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.692681    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 05:44:50.692708 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.692692    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 05:44:50.693120 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.693100    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 05:44:50.693225 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.693162    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-190.us-west-1.compute.internal\" not found"
Nov 01 05:44:50.743466 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.743415    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 05:44:50.745530 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.744718    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 05:44:50.745530 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.744738    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 05:44:50.745530 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.744754    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 05:44:50.745530 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.744791    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 05:44:50.762684 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.762657    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-190.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 05:44:50.762684 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.762665    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-190.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 05:44:50.763555 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.762669    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-190.us-west-1.compute.internal.18e3d58fe4b64837  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-190.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-190.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:44:50.617780279 +0000 UTC m=+0.314939025,LastTimestamp:2026-11-01 05:44:50.617780279 +0000 UTC m=+0.314939025,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-190.us-west-1.compute.internal,}"
Nov 01 05:44:50.793504 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.793483    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:50.794188 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.794173    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:50.794246 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.794203    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:50.794246 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.794214    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:50.794246 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.794237    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:50.845255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.845206    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"]
Nov 01 05:44:50.845330 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.845262    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:50.845997 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.845982    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:50.846061 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.846025    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:50.846096 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.846062    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:50.848159 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848147    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:50.848209 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848194    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 05:44:50.848262 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848250    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bcspm"
Nov 01 05:44:50.848319 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848305    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:50.848356 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848339    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:50.848861 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848839    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:50.848934 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848858    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:50.848934 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848886    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:50.848934 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848896    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:50.848934 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848867    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:50.849089 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.848948    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:50.850829 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.850810    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:50.850917 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.850836    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:50.851563 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.851544    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:50.851637 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.851579    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:50.851637 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.851593    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:50.878792 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.878771    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-190.us-west-1.compute.internal\" not found" node="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:50.884817 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:50.884801    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-190.us-west-1.compute.internal\" not found" node="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:50.924215 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.924194    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:50.926392 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.926367    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:50.926472 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.926400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:50.926472 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.926428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:50.926556 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:50.926488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/784c45e8dd43338b3f3bbbd833ce9232-config\") pod \"kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"784c45e8dd43338b3f3bbbd833ce9232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.027638 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.027615    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:51.027799 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.027787    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:51.029461 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.029431    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.029519 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.029482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.029519 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.029511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/784c45e8dd43338b3f3bbbd833ce9232-config\") pod \"kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"784c45e8dd43338b3f3bbbd833ce9232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.029605 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.029531    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.029605 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.029559    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/784c45e8dd43338b3f3bbbd833ce9232-config\") pod \"kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"784c45e8dd43338b3f3bbbd833ce9232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.029605 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.029568    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.029704 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.029659    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.029704 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:51.029677    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-190.us-west-1.compute.internal\": node \"ip-10-0-138-190.us-west-1.compute.internal\" not found"
Nov 01 05:44:51.123940 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.123874    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.181881 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.181854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.183480 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.183464    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:44:51.184178 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.184164    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.187187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.187168    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:44:51.244852 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.244833    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:44:51.566297 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.566270    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 05:44:51.566809 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.566394    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:44:51.566809 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.566431    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:44:51.566809 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.566465    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:44:51.566809 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:51.566478    2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-190.us-west-1.compute.internal.18e3d58fe6658640\": read tcp 10.0.138.190:36072->3.151.244.222:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-190.us-west-1.compute.internal.18e3d58fe6658640  default   5244 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-190.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-190.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-190.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:44:50 +0000 UTC,LastTimestamp:2026-11-01 05:44:50.794218577 +0000 UTC m=+0.491377324,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-190.us-west-1.compute.internal,}"
Nov 01 05:44:51.604216 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.604194    2575 apiserver.go:52] "Watching apiserver"
Nov 01 05:44:51.624718 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.624700    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 05:44:51.853092 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.853000    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 05:39:52 +0000 UTC" deadline="2028-05-28 15:29:41.741956339 +0000 UTC"
Nov 01 05:44:51.853092 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.853044    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13785h44m49.888915339s"
Nov 01 05:44:51.873463 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.873435    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 05:44:51.873758 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.873739    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-cfp6l","openshift-network-operator/iptables-alerter-cxzmq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4","openshift-cluster-node-tuning-operator/tuned-zcx6z","openshift-network-diagnostics/network-check-target-4gj86","openshift-ovn-kubernetes/ovnkube-node-gdgnj","kube-system/global-pull-secret-syncer-r5qzs","kube-system/konnectivity-agent-jmphx","openshift-dns/node-resolver-gzvr2","openshift-multus/multus-lr295","kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-hh9pk","openshift-image-registry/node-ca-w7rg6"]
Nov 01 05:44:51.876083 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.876068    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:51.878068 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.878050    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:51.880076 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.880062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:51.882044 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.882027    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.884044 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.884029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:51.884104 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:51.884088    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:44:51.886079 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.886065    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.888165 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.888143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:51.888226 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:51.888194    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:44:51.890216 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.890201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:44:51.892336 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.892319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:51.894553 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.894536    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.896791 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.896765    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:51.896862 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:51.896830    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:44:51.898945 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.898930    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:51.926835 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.926607    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 05:44:51.928012 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.927993    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2d6d\""
Nov 01 05:44:51.928167 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.928152    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 05:44:51.928167 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.928159    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 05:44:51.928309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.928159    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 05:44:51.928309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.928195    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 05:44:51.928309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.928240    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 05:44:51.929509 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.929494    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:44:51.929670 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.929658    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzphk\""
Nov 01 05:44:51.929723 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.929711    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 05:44:51.929758 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.929734    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 05:44:51.931569 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.931554    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 05:44:51.931655 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.931644    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 05:44:51.931689 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.931643    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-695z8\""
Nov 01 05:44:51.931689 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.931652    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 05:44:51.934819 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5hf\" (UniqueName: \"kubernetes.io/projected/0593c2e4-1a34-445b-8d91-35a1b0888afe-kube-api-access-gr5hf\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.934862 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-iptables-alerter-script\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:51.934862 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-socket-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:51.934920 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-tmp\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.934920 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-env-overrides\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.934980 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-netns\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.934980 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrcdv\" (UniqueName: \"kubernetes.io/projected/3b530358-dea0-471a-85d7-73749d6fa3af-kube-api-access-jrcdv\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:51.935036 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-modprobe-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935036 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.934997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-kubernetes\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935091 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935091 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-conf\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935091 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935085    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16f463e7-7075-4621-95ae-60c2fefdb950-ovn-node-metrics-cert\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935173 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11dab409-45c8-49c5-9300-24db5bfa7a9f-hosts-file\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:51.935173 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-hostroot\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.935173 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysconfig\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935172    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-sys\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-var-lib-kubelet\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935233    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-systemd-units\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-bin\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935366 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xlr\" (UniqueName: \"kubernetes.io/projected/16f463e7-7075-4621-95ae-60c2fefdb950-kube-api-access-d8xlr\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935366 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935299    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-conf-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.935366 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-slash\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935366 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-netns\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935366 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-var-lib-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-systemd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-bin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.935525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935444    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b530358-dea0-471a-85d7-73749d6fa3af-host\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:51.935525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qg9l\" (UniqueName: \"kubernetes.io/projected/e72e536d-221c-43a0-9a31-0062010750ed-kube-api-access-2qg9l\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:51.935659 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935532    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-etc-tuned\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935659 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:51.935659 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-lib-modules\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.935659 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935604    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:51.935659 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935632    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:51.935796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-cni-binary-copy\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.935796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-socket-dir-parent\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.935796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935697    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-etc-kubernetes\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.935796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-sys-fs\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:51.935796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-system-cni-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:51.935796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935783    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct72m\" (UniqueName: \"kubernetes.io/projected/667231b8-a7ce-45ce-adc5-61b1e84c5f12-kube-api-access-ct72m\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-multus\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935832    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-log-socket\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fww\" (UniqueName: \"kubernetes.io/projected/639fd403-7cfd-438d-90cb-07c383b61678-kube-api-access-k8fww\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935905    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-node-log\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935920    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-kubelet-config\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-cnibin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.935955 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-host-slash\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-host\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.935990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-config\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-dbus\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936031    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskrb\" (UniqueName: \"kubernetes.io/projected/11dab409-45c8-49c5-9300-24db5bfa7a9f-kube-api-access-tskrb\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cnibin\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-daemon-config\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-kubelet\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-run\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5lj\" (UniqueName: \"kubernetes.io/projected/6ee449a4-59d0-40a5-af37-297952887cc3-kube-api-access-gq5lj\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.936187 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936201    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-script-lib\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936219    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-agent-certs\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-system-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936271    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7kx\" (UniqueName: \"kubernetes.io/projected/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-kube-api-access-vs7kx\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936292    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-ovn\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-konnectivity-ca\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-multus-certs\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936424    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-device-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-systemd\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936481    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-etc-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-netd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:51.936524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11dab409-45c8-49c5-9300-24db5bfa7a9f-tmp-dir\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:51.936929 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-kubelet\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.936929 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b530358-dea0-471a-85d7-73749d6fa3af-serviceca\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:51.936929 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-registration-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:51.936929 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:51.936929 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:51.936669    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-k8s-cni-cncf-io\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.025350 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.025318    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 05:44:52.025461 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.025391    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bs7bp"
Nov 01 05:44:52.029522 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.029502    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:44:52.029869 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.029852    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 05:44:52.030122 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030107    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 05:44:52.030220 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030146    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 05:44:52.030220 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030171    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 05:44:52.030220 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030172    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 05:44:52.030220 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030192    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlcv8\""
Nov 01 05:44:52.030220 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030195    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 05:44:52.030494 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030315    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 05:44:52.030494 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030361    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 05:44:52.030494 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030381    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 05:44:52.030494 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030385    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rpsgx\""
Nov 01 05:44:52.030494 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030411    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 05:44:52.030494 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030470    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 05:44:52.030494 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030491    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 05:44:52.030829 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030503    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rz4ht\""
Nov 01 05:44:52.030829 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030535    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 05:44:52.030829 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030549    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qfzvw\""
Nov 01 05:44:52.030829 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030551    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tjqf\""
Nov 01 05:44:52.030829 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030575    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbc6d\""
Nov 01 05:44:52.030829 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.030584    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 05:44:52.037710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskrb\" (UniqueName: \"kubernetes.io/projected/11dab409-45c8-49c5-9300-24db5bfa7a9f-kube-api-access-tskrb\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:52.037778 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cnibin\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.037778 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-daemon-config\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.037778 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-kubelet\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.037778 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.037972 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037808    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cnibin\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.037972 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037839    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-kubelet\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.037972 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037851    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-run\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.037972 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5lj\" (UniqueName: \"kubernetes.io/projected/6ee449a4-59d0-40a5-af37-297952887cc3-kube-api-access-gq5lj\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.037972 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037886    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.037972 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037927    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.037972 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037947    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-run\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.037972 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-script-lib\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037980    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.037997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-agent-certs\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-system-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038071    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-system-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7kx\" (UniqueName: \"kubernetes.io/projected/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-kube-api-access-vs7kx\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-ovn\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-konnectivity-ca\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038204    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-ovn\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-multus-certs\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038287    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-multus-certs\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.038309 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038299    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038339    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-daemon-config\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-device-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-systemd\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038409    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-etc-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038417    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-device-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038422    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038431    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-netd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038480    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-netd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11dab409-45c8-49c5-9300-24db5bfa7a9f-tmp-dir\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038498    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-systemd\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-kubelet\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b530358-dea0-471a-85d7-73749d6fa3af-serviceca\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-registration-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038604    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-k8s-cni-cncf-io\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.038842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5hf\" (UniqueName: \"kubernetes.io/projected/0593c2e4-1a34-445b-8d91-35a1b0888afe-kube-api-access-gr5hf\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-iptables-alerter-script\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038702    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11dab409-45c8-49c5-9300-24db5bfa7a9f-tmp-dir\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038812    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-registration-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038833    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-kubelet\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.038517    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-etc-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.038929    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039055    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039085    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-k8s-cni-cncf-io\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039131    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039153    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039205    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b530358-dea0-471a-85d7-73749d6fa3af-serviceca\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-socket-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039206    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-konnectivity-ca\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.039245    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs podName:639fd403-7cfd-438d-90cb-07c383b61678 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:52.539228237 +0000 UTC m=+2.236386971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs") pod "network-metrics-daemon-hh9pk" (UID: "639fd403-7cfd-438d-90cb-07c383b61678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-tmp\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-env-overrides\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-netns\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.039796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039342    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-socket-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrcdv\" (UniqueName: \"kubernetes.io/projected/3b530358-dea0-471a-85d7-73749d6fa3af-kube-api-access-jrcdv\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039347    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-netns\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-modprobe-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-kubernetes\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039487    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-modprobe-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039501    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-kubernetes\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-conf\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16f463e7-7075-4621-95ae-60c2fefdb950-ovn-node-metrics-cert\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11dab409-45c8-49c5-9300-24db5bfa7a9f-hosts-file\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039654    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11dab409-45c8-49c5-9300-24db5bfa7a9f-hosts-file\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039654    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039706    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-env-overrides\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039712    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-conf\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039717    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-iptables-alerter-script\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-hostroot\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysconfig\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.040680 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039793    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-sys\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-var-lib-kubelet\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-systemd-units\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039867    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysconfig\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-bin\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xlr\" (UniqueName: \"kubernetes.io/projected/16f463e7-7075-4621-95ae-60c2fefdb950-kube-api-access-d8xlr\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039928    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-var-lib-kubelet\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-conf-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039967    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-conf-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-slash\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039987    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-slash\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-netns\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039980    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-bin\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-var-lib-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040054    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-systemd-units\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.039931    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-hostroot\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040089    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-systemd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040155    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-var-lib-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.041525 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040201    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-sys\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040246    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-netns\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-systemd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040389    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-bin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b530358-dea0-471a-85d7-73749d6fa3af-host\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qg9l\" (UniqueName: \"kubernetes.io/projected/e72e536d-221c-43a0-9a31-0062010750ed-kube-api-access-2qg9l\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-etc-tuned\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040487    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040502    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b530358-dea0-471a-85d7-73749d6fa3af-host\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-lib-modules\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-cni-binary-copy\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-socket-dir-parent\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-etc-kubernetes\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-sys-fs\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040685    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-system-cni-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct72m\" (UniqueName: \"kubernetes.io/projected/667231b8-a7ce-45ce-adc5-61b1e84c5f12-kube-api-access-ct72m\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-multus\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-log-socket\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fww\" (UniqueName: \"kubernetes.io/projected/639fd403-7cfd-438d-90cb-07c383b61678-kube-api-access-k8fww\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-node-log\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-kubelet-config\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040919    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-bin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-cnibin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-host-slash\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041543    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041016    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-socket-dir-parent\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041051    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-etc-kubernetes\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041035    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-sys-fs\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.042846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041094    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-system-cni-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041283    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040986    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-multus\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041286    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-node-log\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041326    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-kubelet-config\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041362    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.041397    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.041686    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret podName:3bf4eb38-7ce4-45fe-8c38-0d8907a8861f nodeName:}" failed. No retries permitted until 2026-11-01 05:44:52.541669946 +0000 UTC m=+2.238828681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret") pod "global-pull-secret-syncer-r5qzs" (UID: "3bf4eb38-7ce4-45fe-8c38-0d8907a8861f") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-host\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041701    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-cni-binary-copy\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.040974    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-cnibin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041399    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-log-socket\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041685    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-host-slash\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041478    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-lib-modules\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041749    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-config\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.043527 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-dbus\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:52.044005 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041882    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-dbus\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:52.044005 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.041593    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-host\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.044005 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.042967    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-config\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.044005 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.043029    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-tmp\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.044005 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.043139    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16f463e7-7075-4621-95ae-60c2fefdb950-ovn-node-metrics-cert\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.044005 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.043265    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-agent-certs\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:44:52.044005 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.043783    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-etc-tuned\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.049117 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.049101    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-script-lib\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.099572 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.099539    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7kx\" (UniqueName: \"kubernetes.io/projected/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-kube-api-access-vs7kx\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:52.099658 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.099637    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:52.099658 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.099656    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:52.099769 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.099679    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bwm65 for pod openshift-network-diagnostics/network-check-target-4gj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:52.099769 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.099745    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65 podName:e6ef7b0f-d31d-41bf-a91e-06c64350ec49 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:52.599720483 +0000 UTC m=+2.296879233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bwm65" (UniqueName: "kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65") pod "network-check-target-4gj86" (UID: "e6ef7b0f-d31d-41bf-a91e-06c64350ec49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:52.100009 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.099990    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskrb\" (UniqueName: \"kubernetes.io/projected/11dab409-45c8-49c5-9300-24db5bfa7a9f-kube-api-access-tskrb\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:52.100197 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.100168    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5lj\" (UniqueName: \"kubernetes.io/projected/6ee449a4-59d0-40a5-af37-297952887cc3-kube-api-access-gq5lj\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.100682 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.100661    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5hf\" (UniqueName: \"kubernetes.io/projected/0593c2e4-1a34-445b-8d91-35a1b0888afe-kube-api-access-gr5hf\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.100782 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.100660    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrcdv\" (UniqueName: \"kubernetes.io/projected/3b530358-dea0-471a-85d7-73749d6fa3af-kube-api-access-jrcdv\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:52.101292 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.101268    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xlr\" (UniqueName: \"kubernetes.io/projected/16f463e7-7075-4621-95ae-60c2fefdb950-kube-api-access-d8xlr\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.103497 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.103412    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fww\" (UniqueName: \"kubernetes.io/projected/639fd403-7cfd-438d-90cb-07c383b61678-kube-api-access-k8fww\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:52.103912 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.103886    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qg9l\" (UniqueName: \"kubernetes.io/projected/e72e536d-221c-43a0-9a31-0062010750ed-kube-api-access-2qg9l\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.105144 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.105124    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct72m\" (UniqueName: \"kubernetes.io/projected/667231b8-a7ce-45ce-adc5-61b1e84c5f12-kube-api-access-ct72m\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.213640 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.213610    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:44:52.224937 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.224917    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:44:52.235867 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.235841    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:44:52.248701 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.248662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:44:52.268534 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.268495    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:44:52.275182 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.275157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:44:52.287597 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.287578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:44:52.302217 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.302199    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lr295"
Nov 01 05:44:52.306601 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.306585    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:44:52.344370 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:52.344346    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346b48939138ebf5d66bd099d04ae409.slice/crio-8fdc637a494aa8ee0816ca41c7c57c4aa15fe98004de388c0bcd81710cabd828 WatchSource:0}: Error finding container 8fdc637a494aa8ee0816ca41c7c57c4aa15fe98004de388c0bcd81710cabd828: Status 404 returned error can't find the container with id 8fdc637a494aa8ee0816ca41c7c57c4aa15fe98004de388c0bcd81710cabd828
Nov 01 05:44:52.344791 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:52.344778    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784c45e8dd43338b3f3bbbd833ce9232.slice/crio-378300804cf7bf6f8296483feb9326b50e13fb9ce944d1d4cf49317be9ffc314 WatchSource:0}: Error finding container 378300804cf7bf6f8296483feb9326b50e13fb9ce944d1d4cf49317be9ffc314: Status 404 returned error can't find the container with id 378300804cf7bf6f8296483feb9326b50e13fb9ce944d1d4cf49317be9ffc314
Nov 01 05:44:52.349816 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.349802    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:44:52.546224 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.546195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:52.546372 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.546229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:52.546372 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.546327    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:52.546372 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.546328    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:52.546372 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.546374    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs podName:639fd403-7cfd-438d-90cb-07c383b61678 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:53.54636187 +0000 UTC m=+3.243520604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs") pod "network-metrics-daemon-hh9pk" (UID: "639fd403-7cfd-438d-90cb-07c383b61678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:52.546544 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.546386    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret podName:3bf4eb38-7ce4-45fe-8c38-0d8907a8861f nodeName:}" failed. No retries permitted until 2026-11-01 05:44:53.546380536 +0000 UTC m=+3.243539270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret") pod "global-pull-secret-syncer-r5qzs" (UID: "3bf4eb38-7ce4-45fe-8c38-0d8907a8861f") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:52.646991 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:52.646962    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:52.647608 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.647110    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:52.647608 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.647131    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:52.647608 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.647139    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bwm65 for pod openshift-network-diagnostics/network-check-target-4gj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:52.647608 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:52.647190    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65 podName:e6ef7b0f-d31d-41bf-a91e-06c64350ec49 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:53.647174624 +0000 UTC m=+3.344333377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwm65" (UniqueName: "kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65") pod "network-check-target-4gj86" (UID: "e6ef7b0f-d31d-41bf-a91e-06c64350ec49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:54.009260 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.009232    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:54.049209 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.049160    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"346b48939138ebf5d66bd099d04ae409","Type":"ContainerStarted","Data":"8fdc637a494aa8ee0816ca41c7c57c4aa15fe98004de388c0bcd81710cabd828"}
Nov 01 05:44:54.050128 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.050106    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"784c45e8dd43338b3f3bbbd833ce9232","Type":"ContainerStarted","Data":"378300804cf7bf6f8296483feb9326b50e13fb9ce944d1d4cf49317be9ffc314"}
Nov 01 05:44:54.329357 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.329321    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:53 +0000 UTC" deadline="2028-06-04 12:40:12.050207932 +0000 UTC"
Nov 01 05:44:54.329357 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.329353    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13950h55m17.720858345s"
Nov 01 05:44:54.356239 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:54.356213    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c402d6b_1936_4cea_a20b_42e4a2ec19a9.slice/crio-2d303c776eb7055a0791497d4ffc65a788f07a980e5273a10704e1d90d273243 WatchSource:0}: Error finding container 2d303c776eb7055a0791497d4ffc65a788f07a980e5273a10704e1d90d273243: Status 404 returned error can't find the container with id 2d303c776eb7055a0791497d4ffc65a788f07a980e5273a10704e1d90d273243
Nov 01 05:44:54.422662 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.422635    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:54.445761 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:54.445733    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dab409_45c8_49c5_9300_24db5bfa7a9f.slice/crio-99bc589564b993e0e97b22eeabd99d2220bd7b1de7c6fcce8e0bed144c380b7e WatchSource:0}: Error finding container 99bc589564b993e0e97b22eeabd99d2220bd7b1de7c6fcce8e0bed144c380b7e: Status 404 returned error can't find the container with id 99bc589564b993e0e97b22eeabd99d2220bd7b1de7c6fcce8e0bed144c380b7e
Nov 01 05:44:54.449486 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:54.449468    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f463e7_7075_4621_95ae_60c2fefdb950.slice/crio-dccf9082c6e01ffa488c5481609e9b4c605e685a43664984bea03d90d7e755de WatchSource:0}: Error finding container dccf9082c6e01ffa488c5481609e9b4c605e685a43664984bea03d90d7e755de: Status 404 returned error can't find the container with id dccf9082c6e01ffa488c5481609e9b4c605e685a43664984bea03d90d7e755de
Nov 01 05:44:54.450879 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.450762    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:54.461810 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:54.461790    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667231b8_a7ce_45ce_adc5_61b1e84c5f12.slice/crio-6e272943af250680ae24b73a8ade0f01216b9f3666177b1f23e5a3f7ef70ad6f WatchSource:0}: Error finding container 6e272943af250680ae24b73a8ade0f01216b9f3666177b1f23e5a3f7ef70ad6f: Status 404 returned error can't find the container with id 6e272943af250680ae24b73a8ade0f01216b9f3666177b1f23e5a3f7ef70ad6f
Nov 01 05:44:54.464949 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:54.464928    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bf1b5b_01e1_4f37_b0a9_67a64366ecca.slice/crio-8c0a22b4970447cc1a5f07f5fdbdd042131dcc9592ca0459b50b47223e97894e WatchSource:0}: Error finding container 8c0a22b4970447cc1a5f07f5fdbdd042131dcc9592ca0459b50b47223e97894e: Status 404 returned error can't find the container with id 8c0a22b4970447cc1a5f07f5fdbdd042131dcc9592ca0459b50b47223e97894e
Nov 01 05:44:54.470356 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:54.470337    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b530358_dea0_471a_85d7_73749d6fa3af.slice/crio-3ab6e90da6d95c8112eaeda37c694f31a99222794269476058f8f2bbe1294031 WatchSource:0}: Error finding container 3ab6e90da6d95c8112eaeda37c694f31a99222794269476058f8f2bbe1294031: Status 404 returned error can't find the container with id 3ab6e90da6d95c8112eaeda37c694f31a99222794269476058f8f2bbe1294031
Nov 01 05:44:54.476681 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:54.476653    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee449a4_59d0_40a5_af37_297952887cc3.slice/crio-632f3e652415cbdc6dc8c507bb360e80bfe393e51812ccd2fcc548d6ee2cf854 WatchSource:0}: Error finding container 632f3e652415cbdc6dc8c507bb360e80bfe393e51812ccd2fcc548d6ee2cf854: Status 404 returned error can't find the container with id 632f3e652415cbdc6dc8c507bb360e80bfe393e51812ccd2fcc548d6ee2cf854
Nov 01 05:44:54.502974 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:44:54.502953    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72e536d_221c_43a0_9a31_0062010750ed.slice/crio-6540a00891483588ac51739dc2b762ae97b83227d69bd9cf538a1b950193d17b WatchSource:0}: Error finding container 6540a00891483588ac51739dc2b762ae97b83227d69bd9cf538a1b950193d17b: Status 404 returned error can't find the container with id 6540a00891483588ac51739dc2b762ae97b83227d69bd9cf538a1b950193d17b
Nov 01 05:44:54.853741 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.853703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:54.853741 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.853746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:54.854005 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:54.853886    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:54.854005 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:54.853939    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs podName:639fd403-7cfd-438d-90cb-07c383b61678 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:56.853925797 +0000 UTC m=+5.250707176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs") pod "network-metrics-daemon-hh9pk" (UID: "639fd403-7cfd-438d-90cb-07c383b61678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:54.854005 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:54.854004    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:54.854179 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:54.854030    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret podName:3bf4eb38-7ce4-45fe-8c38-0d8907a8861f nodeName:}" failed. No retries permitted until 2026-11-01 05:44:56.854023116 +0000 UTC m=+5.250804494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret") pod "global-pull-secret-syncer-r5qzs" (UID: "3bf4eb38-7ce4-45fe-8c38-0d8907a8861f") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:54.954264 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:54.954230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:54.954754 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:54.954361    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:54.954754 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:54.954377    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:54.954754 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:54.954386    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bwm65 for pod openshift-network-diagnostics/network-check-target-4gj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:54.954754 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:54.954429    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65 podName:e6ef7b0f-d31d-41bf-a91e-06c64350ec49 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:56.954416741 +0000 UTC m=+5.351198115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwm65" (UniqueName: "kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65") pod "network-check-target-4gj86" (UID: "e6ef7b0f-d31d-41bf-a91e-06c64350ec49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:55.046460 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.046429    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:55.046460 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.046450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:55.046684 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:55.046568    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:44:55.046684 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.046578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:55.046684 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:55.046657    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:44:55.046851 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:55.046739    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:44:55.053093 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.053058    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzvr2" event={"ID":"11dab409-45c8-49c5-9300-24db5bfa7a9f","Type":"ContainerStarted","Data":"99bc589564b993e0e97b22eeabd99d2220bd7b1de7c6fcce8e0bed144c380b7e"}
Nov 01 05:44:55.054653 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.054494    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cxzmq" event={"ID":"2c402d6b-1936-4cea-a20b-42e4a2ec19a9","Type":"ContainerStarted","Data":"2d303c776eb7055a0791497d4ffc65a788f07a980e5273a10704e1d90d273243"}
Nov 01 05:44:55.056312 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.056287    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zcx6z" event={"ID":"6ee449a4-59d0-40a5-af37-297952887cc3","Type":"ContainerStarted","Data":"632f3e652415cbdc6dc8c507bb360e80bfe393e51812ccd2fcc548d6ee2cf854"}
Nov 01 05:44:55.057618 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.057595    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w7rg6" event={"ID":"3b530358-dea0-471a-85d7-73749d6fa3af","Type":"ContainerStarted","Data":"3ab6e90da6d95c8112eaeda37c694f31a99222794269476058f8f2bbe1294031"}
Nov 01 05:44:55.058698 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.058670    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jmphx" event={"ID":"76bf1b5b-01e1-4f37-b0a9-67a64366ecca","Type":"ContainerStarted","Data":"8c0a22b4970447cc1a5f07f5fdbdd042131dcc9592ca0459b50b47223e97894e"}
Nov 01 05:44:55.059976 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.059955    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" event={"ID":"667231b8-a7ce-45ce-adc5-61b1e84c5f12","Type":"ContainerStarted","Data":"6e272943af250680ae24b73a8ade0f01216b9f3666177b1f23e5a3f7ef70ad6f"}
Nov 01 05:44:55.062347 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.062271    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"dccf9082c6e01ffa488c5481609e9b4c605e685a43664984bea03d90d7e755de"}
Nov 01 05:44:55.063684 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.063661    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4" event={"ID":"e72e536d-221c-43a0-9a31-0062010750ed","Type":"ContainerStarted","Data":"6540a00891483588ac51739dc2b762ae97b83227d69bd9cf538a1b950193d17b"}
Nov 01 05:44:55.065555 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.065527    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lr295" event={"ID":"0593c2e4-1a34-445b-8d91-35a1b0888afe","Type":"ContainerStarted","Data":"6136a52e6139f67c300dd3518fe7f9abca2f66b15db5003dfdc39054036638c8"}
Nov 01 05:44:55.330539 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.330475    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:53 +0000 UTC" deadline="2028-08-08 13:17:42.25347573 +0000 UTC"
Nov 01 05:44:55.330539 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:55.330538    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15511h32m46.9229416s"
Nov 01 05:44:56.868207 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:56.868168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:56.868652 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:56.868223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:56.868652 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:56.868340    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:56.868652 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:56.868405    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs podName:639fd403-7cfd-438d-90cb-07c383b61678 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:00.868387806 +0000 UTC m=+9.265169198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs") pod "network-metrics-daemon-hh9pk" (UID: "639fd403-7cfd-438d-90cb-07c383b61678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:56.868652 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:56.868469    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:56.868652 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:56.868505    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret podName:3bf4eb38-7ce4-45fe-8c38-0d8907a8861f nodeName:}" failed. No retries permitted until 2026-11-01 05:45:00.868493966 +0000 UTC m=+9.265275356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret") pod "global-pull-secret-syncer-r5qzs" (UID: "3bf4eb38-7ce4-45fe-8c38-0d8907a8861f") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:56.968701 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:56.968664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:56.968888 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:56.968848    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:56.968888 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:56.968871    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:56.968888 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:56.968882    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bwm65 for pod openshift-network-diagnostics/network-check-target-4gj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:56.969057 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:56.968936    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65 podName:e6ef7b0f-d31d-41bf-a91e-06c64350ec49 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:00.968919666 +0000 UTC m=+9.365701058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwm65" (UniqueName: "kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65") pod "network-check-target-4gj86" (UID: "e6ef7b0f-d31d-41bf-a91e-06c64350ec49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:57.045510 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:57.045486    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:57.045686 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:57.045661    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:44:57.045761 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:57.045727    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:57.045815 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:57.045780    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:44:57.045815 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:57.045811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:57.045957 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:57.045893    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:44:58.100530 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:58.100448    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"784c45e8dd43338b3f3bbbd833ce9232","Type":"ContainerStarted","Data":"00e1cf3d6069e1b0e0c5c0d01bb926ca13ade8ca5b262d9aa0ec6cb56bef48ec"}
Nov 01 05:44:58.108097 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:58.108037    2575 generic.go:356] "Generic (PLEG): container finished" podID="346b48939138ebf5d66bd099d04ae409" containerID="030639c3a91d14d2d6568cb610ee3b1cc6eaede499039fdc88747e372f7e5a70" exitCode=0
Nov 01 05:44:58.108227 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:58.108168    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"346b48939138ebf5d66bd099d04ae409","Type":"ContainerDied","Data":"030639c3a91d14d2d6568cb610ee3b1cc6eaede499039fdc88747e372f7e5a70"}
Nov 01 05:44:58.117302 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:58.117125    2575 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="5f9db22f50c14387c880f80c8ea4807bbe49719d4c66ed0e43883a4ecd6a0f45" exitCode=0
Nov 01 05:44:58.117302 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:58.117165    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" event={"ID":"667231b8-a7ce-45ce-adc5-61b1e84c5f12","Type":"ContainerDied","Data":"5f9db22f50c14387c880f80c8ea4807bbe49719d4c66ed0e43883a4ecd6a0f45"}
Nov 01 05:44:58.327929 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:58.326678    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal" podStartSLOduration=6.326657583 podStartE2EDuration="6.326657583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:58.213231764 +0000 UTC m=+6.610013162" watchObservedRunningTime="2026-11-01 05:44:58.326657583 +0000 UTC m=+6.723438980"
Nov 01 05:44:59.045522 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:59.045501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:44:59.045628 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:59.045607    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:44:59.045694 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:59.045659    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:44:59.045753 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:59.045708    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:44:59.045811 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:59.045781    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:44:59.045909 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:44:59.045887    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:44:59.122294 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:44:59.122258    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"346b48939138ebf5d66bd099d04ae409","Type":"ContainerStarted","Data":"4657dbccc01a359ad24ee2b07a81ed1cb44543d70a6a9f3926e378bb1f42605d"}
Nov 01 05:45:00.147910 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:00.147870    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jmphx" event={"ID":"76bf1b5b-01e1-4f37-b0a9-67a64366ecca","Type":"ContainerStarted","Data":"217b688dd719ed806019a8f3004af6313f3d0ae1dcefd6a834c29144cbd51b53"}
Nov 01 05:45:00.263053 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:00.262982    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal" podStartSLOduration=8.262963529 podStartE2EDuration="8.262963529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:59.236812421 +0000 UTC m=+7.633593838" watchObservedRunningTime="2026-11-01 05:45:00.262963529 +0000 UTC m=+8.659744926"
Nov 01 05:45:00.367112 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:00.366935    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:45:00.368086 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:00.367902    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:45:00.481871 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:00.481280    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jmphx" podStartSLOduration=3.92380203 podStartE2EDuration="8.481262618s" totalImagesPullingTime="4.557460588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:00.2624836 +0000 UTC m=+8.659264997" watchObservedRunningTime="2026-11-01 05:45:00.481262618 +0000 UTC m=+8.878044015"
Nov 01 05:45:00.905725 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:00.905694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:00.905879 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:00.905755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:00.905946 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:00.905885    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:00.905946 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:00.905931    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs podName:639fd403-7cfd-438d-90cb-07c383b61678 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:08.905919401 +0000 UTC m=+17.302700789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs") pod "network-metrics-daemon-hh9pk" (UID: "639fd403-7cfd-438d-90cb-07c383b61678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:00.906067 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:00.905975    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:00.906067 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:00.906004    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret podName:3bf4eb38-7ce4-45fe-8c38-0d8907a8861f nodeName:}" failed. No retries permitted until 2026-11-01 05:45:08.905995013 +0000 UTC m=+17.302776402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret") pod "global-pull-secret-syncer-r5qzs" (UID: "3bf4eb38-7ce4-45fe-8c38-0d8907a8861f") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:01.006388 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.006314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:01.006540 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:01.006499    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:45:01.006540 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:01.006519    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:45:01.006540 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:01.006532    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bwm65 for pod openshift-network-diagnostics/network-check-target-4gj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:01.006656 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:01.006588    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65 podName:e6ef7b0f-d31d-41bf-a91e-06c64350ec49 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:09.006570615 +0000 UTC m=+17.403352013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwm65" (UniqueName: "kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65") pod "network-check-target-4gj86" (UID: "e6ef7b0f-d31d-41bf-a91e-06c64350ec49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:01.046237 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.046206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:01.046419 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:01.046317    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:01.046419 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.046375    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:01.046532 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:01.046422    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:01.046532 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.046477    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:01.046532 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:01.046520    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:01.151965 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.151917    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w7rg6" event={"ID":"3b530358-dea0-471a-85d7-73749d6fa3af","Type":"ContainerStarted","Data":"dc674d96042806e47661f0dae6505fc51f747b163c32c817505b0a29c4b8aa9f"}
Nov 01 05:45:01.156647 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.156618    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4" event={"ID":"e72e536d-221c-43a0-9a31-0062010750ed","Type":"ContainerStarted","Data":"5581acff662fea79772a74df822aed554f4d00e651c7ed9f45b763310b4f31ec"}
Nov 01 05:45:01.844517 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.844450    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:45:01.845307 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.845283    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:45:01.959408 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:01.959351    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w7rg6" podStartSLOduration=4.36155346 podStartE2EDuration="9.959331214s" totalImagesPullingTime="5.597777754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:01.26655151 +0000 UTC m=+9.663332905" watchObservedRunningTime="2026-11-01 05:45:01.959331214 +0000 UTC m=+10.356112613"
Nov 01 05:45:02.912237 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:02.912215    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 05:45:02.998846 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:02.998664    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:45:02.912233034Z","UUID":"8850b8b4-bb7e-4739-924c-c68c64c254ba","Handler":null,"Name":"","Endpoint":""}
Nov 01 05:45:03.000360 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.000335    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 05:45:03.000468 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.000371    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 05:45:03.045561 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.045524    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:03.045733 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.045632    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:03.045733 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:03.045646    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:03.045733 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:03.045728    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:03.045907 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.045775    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:03.045907 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:03.045853    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:03.164979 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.164952    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4" event={"ID":"e72e536d-221c-43a0-9a31-0062010750ed","Type":"ContainerStarted","Data":"880352f3e6aad1d6bd167ddfbfabcbec80dd9e4e61a79b854c037ef22b6cc489"}
Nov 01 05:45:03.168308 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.168278    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lr295" event={"ID":"0593c2e4-1a34-445b-8d91-35a1b0888afe","Type":"ContainerStarted","Data":"3905e6f2dc9ae5faca57be48f9211c65d92b8ae82d409830fa0b421192c41a76"}
Nov 01 05:45:03.170838 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.170801    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzvr2" event={"ID":"11dab409-45c8-49c5-9300-24db5bfa7a9f","Type":"ContainerStarted","Data":"e218c9f39e633b2c46b3daae875c09eb96a382c0d6f965de0b110a8c4aff6575"}
Nov 01 05:45:03.399102 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.398885    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lr295" podStartSLOduration=3.557783141 podStartE2EDuration="11.398866082s" totalImagesPullingTime="7.841082941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:03.285442372 +0000 UTC m=+11.682223769" watchObservedRunningTime="2026-11-01 05:45:03.398866082 +0000 UTC m=+11.795647479"
Nov 01 05:45:03.399904 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:03.399865    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gzvr2" podStartSLOduration=3.105151032 podStartE2EDuration="11.399854748s" totalImagesPullingTime="8.294703716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:03.399142386 +0000 UTC m=+11.795923777" watchObservedRunningTime="2026-11-01 05:45:03.399854748 +0000 UTC m=+11.796636146"
Nov 01 05:45:04.175851 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:04.175798    2575 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="894843df3da7b66fd644738cac2b53a228800c69c80c9374d683a7e38d24e74a" exitCode=0
Nov 01 05:45:04.176301 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:04.175919    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" event={"ID":"667231b8-a7ce-45ce-adc5-61b1e84c5f12","Type":"ContainerDied","Data":"894843df3da7b66fd644738cac2b53a228800c69c80c9374d683a7e38d24e74a"}
Nov 01 05:45:04.180396 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:04.180368    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cxzmq" event={"ID":"2c402d6b-1936-4cea-a20b-42e4a2ec19a9","Type":"ContainerStarted","Data":"fa7cc885cd9f2d586ad987df1b09a1c09506861d8bc0a62a00991daad534e75a"}
Nov 01 05:45:04.416531 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:04.416462    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cxzmq" podStartSLOduration=4.360356424 podStartE2EDuration="12.416432279s" totalImagesPullingTime="8.056075855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:04.415936633 +0000 UTC m=+12.812718030" watchObservedRunningTime="2026-11-01 05:45:04.416432279 +0000 UTC m=+12.813213675"
Nov 01 05:45:05.045397 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:05.045358    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:05.045397 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:05.045383    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:05.045649 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:05.045506    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:05.045649 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:05.045580    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:05.045752 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:05.045645    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:05.045752 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:05.045727    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:05.184504 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:05.184473    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4" event={"ID":"e72e536d-221c-43a0-9a31-0062010750ed","Type":"ContainerStarted","Data":"75ce2f7a773437d313ab1a58b8ef4a74e804fcaa7f2be78ca32252c4e88ac10b"}
Nov 01 05:45:05.301453 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:05.301345    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4" podStartSLOduration=3.8015795089999997 podStartE2EDuration="13.301326103s" totalImagesPullingTime="9.499746594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:05.301144265 +0000 UTC m=+13.697925665" watchObservedRunningTime="2026-11-01 05:45:05.301326103 +0000 UTC m=+13.698107499"
Nov 01 05:45:06.188152 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:06.188116    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zcx6z" event={"ID":"6ee449a4-59d0-40a5-af37-297952887cc3","Type":"ContainerStarted","Data":"dfa1ade77cb242249f39a261dd9e716ab499fd2c0e51b13a03613a14177723db"}
Nov 01 05:45:06.190402 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:06.190375    2575 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="b224925d5d5758cb8aca46c47835c9eace8621b0f6d18edf2935a0dad0895b92" exitCode=0
Nov 01 05:45:06.190541 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:06.190460    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" event={"ID":"667231b8-a7ce-45ce-adc5-61b1e84c5f12","Type":"ContainerDied","Data":"b224925d5d5758cb8aca46c47835c9eace8621b0f6d18edf2935a0dad0895b92"}
Nov 01 05:45:06.304540 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:06.304478    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zcx6z" podStartSLOduration=3.654301417 podStartE2EDuration="14.304460771s" totalImagesPullingTime="10.650159354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:06.304299615 +0000 UTC m=+14.701081011" watchObservedRunningTime="2026-11-01 05:45:06.304460771 +0000 UTC m=+14.701242170"
Nov 01 05:45:07.046266 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:07.046228    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:07.046266 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:07.046250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:07.046505 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:07.046360    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:07.046505 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:07.046421    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:07.046599 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:07.046542    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:07.046647 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:07.046633    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:08.197802 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:08.197611    2575 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="e2e1bdca5160e094ced14d30e9e9f80666d700871e6c31206a7314d0e9ee364e" exitCode=0
Nov 01 05:45:08.198368 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:08.197694    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" event={"ID":"667231b8-a7ce-45ce-adc5-61b1e84c5f12","Type":"ContainerDied","Data":"e2e1bdca5160e094ced14d30e9e9f80666d700871e6c31206a7314d0e9ee364e"}
Nov 01 05:45:08.967369 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:08.967321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:08.967524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:08.967441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:08.967524 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:08.967472    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:08.967636 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:08.967553    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs podName:639fd403-7cfd-438d-90cb-07c383b61678 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:24.967533878 +0000 UTC m=+33.364315274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs") pod "network-metrics-daemon-hh9pk" (UID: "639fd403-7cfd-438d-90cb-07c383b61678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:08.967636 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:08.967578    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:08.967636 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:08.967635    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret podName:3bf4eb38-7ce4-45fe-8c38-0d8907a8861f nodeName:}" failed. No retries permitted until 2026-11-01 05:45:24.967618466 +0000 UTC m=+33.364399854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret") pod "global-pull-secret-syncer-r5qzs" (UID: "3bf4eb38-7ce4-45fe-8c38-0d8907a8861f") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:09.045907 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:09.045872    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:09.045907 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:09.045903    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:09.046144 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:09.045870    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:09.046144 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:09.046010    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:09.046144 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:09.046077    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:09.046256 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:09.046147    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:09.068066 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:09.068040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:09.068220 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:09.068177    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:45:09.068220 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:09.068190    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:45:09.068220 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:09.068200    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bwm65 for pod openshift-network-diagnostics/network-check-target-4gj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:09.068342 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:09.068245    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65 podName:e6ef7b0f-d31d-41bf-a91e-06c64350ec49 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:25.068234008 +0000 UTC m=+33.465015382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwm65" (UniqueName: "kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65") pod "network-check-target-4gj86" (UID: "e6ef7b0f-d31d-41bf-a91e-06c64350ec49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:10.206647 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:10.206101    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"3a6ef0b2fa37fc719ea3500ddb30ccbc21a28e2c45822c84092cd68b7273c1f1"}
Nov 01 05:45:10.206647 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:10.206393    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"b2e3125ab8ff6c259ae0633b17972710402fa9205a3fc1ad856c93e42c898b31"}
Nov 01 05:45:10.206647 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:10.206444    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"cb3124dd0713b4dfa573853910dc0b18f2ddf17be40017ad9bf0c6e882e5b2af"}
Nov 01 05:45:10.206647 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:10.206492    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"f104d2692f483063527bd3317ce46adfa815cb5734e63c06280db55b4e62b75e"}
Nov 01 05:45:10.206647 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:10.206540    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"53f79a86783b1af8de5f019d5e5d577b8c8709497038d0ba55641bec48bb17f9"}
Nov 01 05:45:10.206647 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:10.206586    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"ecb4b0a8e832c4622aa5322e485f24c545503a7a24a8e3159d8623c500a6e39d"}
Nov 01 05:45:11.046022 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:11.045980    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:11.046193 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:11.046028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:11.046193 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:11.045986    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:11.046193 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:11.046110    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:11.046193 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:11.046157    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:11.046338 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:11.046226    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:11.211409 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:11.211380    2575 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="af970590b9ccf20931af5592390eb67e89cb7fa3d755a6f476d203cb0f9e6718" exitCode=0
Nov 01 05:45:11.211774 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:11.211430    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" event={"ID":"667231b8-a7ce-45ce-adc5-61b1e84c5f12","Type":"ContainerDied","Data":"af970590b9ccf20931af5592390eb67e89cb7fa3d755a6f476d203cb0f9e6718"}
Nov 01 05:45:12.216760 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:12.216729    2575 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="239f24f6dafbfb65b1725de341ba79c0f74629a81c01b72af1202a0193f89a83" exitCode=0
Nov 01 05:45:12.217320 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:12.216800    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" event={"ID":"667231b8-a7ce-45ce-adc5-61b1e84c5f12","Type":"ContainerDied","Data":"239f24f6dafbfb65b1725de341ba79c0f74629a81c01b72af1202a0193f89a83"}
Nov 01 05:45:12.220522 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:12.220494    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"7e799b1f7c9274968a06b6f010af3658c3cc8d464e4a8d0d5f8dbb01e2a59132"}
Nov 01 05:45:13.045372 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:13.045317    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:13.045583 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:13.045382    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:13.045583 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:13.045341    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:13.045583 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:13.045463    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:13.045583 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:13.045558    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:13.045753 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:13.045636    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:13.226206 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:13.226177    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" event={"ID":"667231b8-a7ce-45ce-adc5-61b1e84c5f12","Type":"ContainerStarted","Data":"28f3f66762f381de5b7d44898b284c74c19156148687776c314ff95dbe758e06"}
Nov 01 05:45:13.347286 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:13.347194    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cfp6l" podStartSLOduration=8.571559196 podStartE2EDuration="21.347180954s" totalImagesPullingTime="12.775621758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:13.346792913 +0000 UTC m=+21.743574320" watchObservedRunningTime="2026-11-01 05:45:13.347180954 +0000 UTC m=+21.743962349"
Nov 01 05:45:14.048645 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:14.048617    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:14.048812 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:14.048717    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:15.046068 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:15.046030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:15.046068 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:15.046058    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:15.046757 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:15.046137    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:15.046757 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:15.046264    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:15.234549 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:15.234504    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" event={"ID":"16f463e7-7075-4621-95ae-60c2fefdb950","Type":"ContainerStarted","Data":"3d0cee9a8fe6c49eec7c129b316625ae7b3347ad4d30cf7ee19ed4bf14d568be"}
Nov 01 05:45:15.234913 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:15.234850    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:45:15.250886 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:15.250862    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:45:15.650420 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:15.650217    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj" podStartSLOduration=8.355732428 podStartE2EDuration="23.650203248s" totalImagesPullingTime="15.29447082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:15.521754192 +0000 UTC m=+23.918535589" watchObservedRunningTime="2026-11-01 05:45:15.650203248 +0000 UTC m=+24.046984644"
Nov 01 05:45:16.049264 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:16.049240    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:16.049634 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:16.049331    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:16.237522 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:16.237493    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:45:16.237734 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:16.237569    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:45:16.253174 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:16.253151    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:45:17.045879 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:17.045852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:17.045879 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:17.045887    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:17.046132 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:17.045967    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:17.046132 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:17.046087    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:17.122321 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:17.122292    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5qzs"]
Nov 01 05:45:17.125240 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:17.125207    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gj86"]
Nov 01 05:45:17.125361 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:17.125310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:17.125431 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:17.125398    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:17.125909 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:17.125887    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hh9pk"]
Nov 01 05:45:17.238809 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:17.238781    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:17.239125 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:17.239109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:17.239223 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:17.239208    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:17.239342 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:17.239318    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:19.045567 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:19.045519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:19.046008 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:19.045528    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:19.046008 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:19.045665    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:19.046008 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:19.045821    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:19.046008 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:19.045869    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:19.046008 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:19.045962    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:21.046119 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:21.046085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:21.046119 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:21.046104    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:21.046119 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:21.046115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:21.046650 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:21.046195    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:21.046650 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:21.046262    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:21.046650 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:21.046321    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:23.046277 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:23.046238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:23.046794 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:23.046249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:23.046794 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:23.046382    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:23.046794 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:23.046382    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:23.046794 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:23.046448    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:23.046794 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:23.046502    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:23.568706 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:23.568678    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzvr2_11dab409-45c8-49c5-9300-24db5bfa7a9f/dns-node-resolver/0.log"
Nov 01 05:45:24.596531 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:24.596504    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w7rg6_3b530358-dea0-471a-85d7-73749d6fa3af/node-ca/0.log"
Nov 01 05:45:24.991280 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:24.991248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:24.991464 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:24.991294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:24.991464 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:24.991396    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:24.991464 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:24.991400    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:24.991464 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:24.991452    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs podName:639fd403-7cfd-438d-90cb-07c383b61678 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:56.991436207 +0000 UTC m=+65.388217601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs") pod "network-metrics-daemon-hh9pk" (UID: "639fd403-7cfd-438d-90cb-07c383b61678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:24.991464 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:24.991466    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret podName:3bf4eb38-7ce4-45fe-8c38-0d8907a8861f nodeName:}" failed. No retries permitted until 2026-11-01 05:45:56.991460434 +0000 UTC m=+65.388241808 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret") pod "global-pull-secret-syncer-r5qzs" (UID: "3bf4eb38-7ce4-45fe-8c38-0d8907a8861f") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:25.046041 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:25.046007    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:25.046041 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:25.046030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:25.046281 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:25.046109    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:25.046281 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:25.046141    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:25.046281 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:25.046241    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:25.046393 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:25.046303    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:25.092446 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:25.092414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:25.092617 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:25.092547    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:45:25.092617 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:25.092568    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:45:25.092617 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:25.092581    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bwm65 for pod openshift-network-diagnostics/network-check-target-4gj86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:25.092752 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:25.092633    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65 podName:e6ef7b0f-d31d-41bf-a91e-06c64350ec49 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:57.092617428 +0000 UTC m=+65.489398812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwm65" (UniqueName: "kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65") pod "network-check-target-4gj86" (UID: "e6ef7b0f-d31d-41bf-a91e-06c64350ec49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:27.045958 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:27.045923    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:27.045958 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:27.045951    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:27.046488 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:27.046015    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:27.046488 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:27.046098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:27.046488 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:27.046206    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:27.046488 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:27.046299    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:29.046353 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:29.046165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:29.046870 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:29.046166    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:29.046870 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:29.046449    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:29.046870 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:29.046165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:29.046870 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:29.046505    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:29.046870 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:29.046567    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:31.046264 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:31.046233    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:31.046729 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:31.046294    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:31.046729 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:31.046384    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:31.046729 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:31.046389    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:31.046729 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:31.046444    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:31.046729 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:31.046564    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:33.046162 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:33.046114    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:33.046668 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:33.046118    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:33.046668 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:33.046216    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:33.046668 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:33.046307    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:33.046668 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:33.046122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:33.046668 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:33.046393    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:35.046304 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:35.046267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:35.046304 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:35.046296    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:35.046686 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:35.046310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:35.046686 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:35.046396    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:35.046686 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:35.046504    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:35.046686 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:35.046563    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:37.046087 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:37.046051    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:37.046087 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:37.046082    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:37.046540 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:37.046153    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:37.046540 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:37.046197    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:37.046540 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:37.046285    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:37.046540 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:37.046377    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:39.045842 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:39.045791    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:39.046327 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:39.045870    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:39.046327 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:39.045897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:39.046327 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:39.046011    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5qzs" podUID="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f"
Nov 01 05:45:39.046327 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:39.046046    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gj86" podUID="e6ef7b0f-d31d-41bf-a91e-06c64350ec49"
Nov 01 05:45:39.046327 ip-10-0-138-190 kubenswrapper[2575]: E1101 05:45:39.046122    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9pk" podUID="639fd403-7cfd-438d-90cb-07c383b61678"
Nov 01 05:45:40.412944 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.412871    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeReady"
Nov 01 05:45:40.413295 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.412992    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 05:45:40.566683 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.566654    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vzzng"]
Nov 01 05:45:40.568468 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.568453    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wbwlt"]
Nov 01 05:45:40.568606 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.568588    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.570190 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.570171    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:45:40.577790 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.577764    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pzwxh"]
Nov 01 05:45:40.579758 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.579740    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vzzng"]
Nov 01 05:45:40.579870 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.579843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.588744 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.588721    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wbwlt"]
Nov 01 05:45:40.620790 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.620764    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 05:45:40.620790 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.620791    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zklw\""
Nov 01 05:45:40.620973 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.620798    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 05:45:40.621491 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.621475    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 05:45:40.621583 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.621572    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 05:45:40.621719 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.621709    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vp9l2\""
Nov 01 05:45:40.628215 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.628196    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pzwxh"]
Nov 01 05:45:40.694894 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.694802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c203e1-de50-4a9e-a77f-1c7a97335ba1-config-volume\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.694894 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.694851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpz5k\" (UniqueName: \"kubernetes.io/projected/7bec7b62-a68f-4579-8bcc-14eb1c26f163-kube-api-access-lpz5k\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:45:40.694894 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.694882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c203e1-de50-4a9e-a77f-1c7a97335ba1-metrics-tls\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.695141 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.694908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpkf\" (UniqueName: \"kubernetes.io/projected/70c203e1-de50-4a9e-a77f-1c7a97335ba1-kube-api-access-qmpkf\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.695141 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.694953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc4f5277-87b1-43e4-a47e-2a42cc73f400-data-volume\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.695141 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.694979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc4f5277-87b1-43e4-a47e-2a42cc73f400-crio-socket\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.695141 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.695003    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcdfb\" (UniqueName: \"kubernetes.io/projected/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-api-access-pcdfb\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.695141 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.695057    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc4f5277-87b1-43e4-a47e-2a42cc73f400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.695337 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.695174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c203e1-de50-4a9e-a77f-1c7a97335ba1-tmp-dir\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.695337 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.695228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.695337 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.695291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bec7b62-a68f-4579-8bcc-14eb1c26f163-cert\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:45:40.720821 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.720786    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 05:45:40.720821 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.720821    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 05:45:40.721040 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.720820    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 05:45:40.721040 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.720915    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 05:45:40.721040 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.720976    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74ptq\""
Nov 01 05:45:40.721040 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.721017    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 05:45:40.795948 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.795913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc4f5277-87b1-43e4-a47e-2a42cc73f400-crio-socket\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.796133 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.795954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdfb\" (UniqueName: \"kubernetes.io/projected/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-api-access-pcdfb\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.796133 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.795982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc4f5277-87b1-43e4-a47e-2a42cc73f400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.796255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796141    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc4f5277-87b1-43e4-a47e-2a42cc73f400-crio-socket\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.796255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c203e1-de50-4a9e-a77f-1c7a97335ba1-tmp-dir\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.796255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.796388 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bec7b62-a68f-4579-8bcc-14eb1c26f163-cert\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:45:40.796388 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c203e1-de50-4a9e-a77f-1c7a97335ba1-config-volume\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.796388 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpz5k\" (UniqueName: \"kubernetes.io/projected/7bec7b62-a68f-4579-8bcc-14eb1c26f163-kube-api-access-lpz5k\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:45:40.796388 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c203e1-de50-4a9e-a77f-1c7a97335ba1-metrics-tls\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.796632 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpkf\" (UniqueName: \"kubernetes.io/projected/70c203e1-de50-4a9e-a77f-1c7a97335ba1-kube-api-access-qmpkf\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.796632 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796426    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc4f5277-87b1-43e4-a47e-2a42cc73f400-data-volume\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.796632 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796530    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c203e1-de50-4a9e-a77f-1c7a97335ba1-tmp-dir\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.796763 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796725    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc4f5277-87b1-43e4-a47e-2a42cc73f400-data-volume\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.796910 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796889    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.796975 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.796919    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c203e1-de50-4a9e-a77f-1c7a97335ba1-config-volume\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.800168 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.800147    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c203e1-de50-4a9e-a77f-1c7a97335ba1-metrics-tls\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.800264 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.800177    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc4f5277-87b1-43e4-a47e-2a42cc73f400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.800264 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.800242    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bec7b62-a68f-4579-8bcc-14eb1c26f163-cert\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:45:40.856627 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.856600    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpkf\" (UniqueName: \"kubernetes.io/projected/70c203e1-de50-4a9e-a77f-1c7a97335ba1-kube-api-access-qmpkf\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.856767 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.856650    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdfb\" (UniqueName: \"kubernetes.io/projected/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-api-access-pcdfb\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:40.856806 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.856796    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpz5k\" (UniqueName: \"kubernetes.io/projected/7bec7b62-a68f-4579-8bcc-14eb1c26f163-kube-api-access-lpz5k\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:45:40.884361 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.884332    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:40.888780 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.888757    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:45:40.896715 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:40.896693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:45:41.046048 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.046014    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:41.046201 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.046137    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:41.046201 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.046023    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:41.098181 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.098154    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 05:45:41.098348 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.098332    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 05:45:41.098396 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.098335    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v8m4f\""
Nov 01 05:45:41.098396 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.098378    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:45:41.098457 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.098449    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4d6z\""
Nov 01 05:45:41.098522 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.098512    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:45:41.098554 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.098549    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87mzr\""
Nov 01 05:45:41.282484 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.282454    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vzzng"]
Nov 01 05:45:41.287598 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:41.287569    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c203e1_de50_4a9e_a77f_1c7a97335ba1.slice/crio-11e82adff7c925facf0db21edd8d03cdbd2e11f58dc1aa9cac049ae47e19a284 WatchSource:0}: Error finding container 11e82adff7c925facf0db21edd8d03cdbd2e11f58dc1aa9cac049ae47e19a284: Status 404 returned error can't find the container with id 11e82adff7c925facf0db21edd8d03cdbd2e11f58dc1aa9cac049ae47e19a284
Nov 01 05:45:41.327441 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.327412    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wbwlt"]
Nov 01 05:45:41.331204 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:41.331085    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bec7b62_a68f_4579_8bcc_14eb1c26f163.slice/crio-7cd95c36d197031d96e0c501a4ca42b34bc5cfc02e3a99bd4a16572278a13990 WatchSource:0}: Error finding container 7cd95c36d197031d96e0c501a4ca42b34bc5cfc02e3a99bd4a16572278a13990: Status 404 returned error can't find the container with id 7cd95c36d197031d96e0c501a4ca42b34bc5cfc02e3a99bd4a16572278a13990
Nov 01 05:45:41.332584 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:41.332468    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pzwxh"]
Nov 01 05:45:41.336398 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:41.336377    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4f5277_87b1_43e4_a47e_2a42cc73f400.slice/crio-fa3635c9725c8992ad54c36ff4dec74304c03a883c3ef4a7259d95c1300bbb60 WatchSource:0}: Error finding container fa3635c9725c8992ad54c36ff4dec74304c03a883c3ef4a7259d95c1300bbb60: Status 404 returned error can't find the container with id fa3635c9725c8992ad54c36ff4dec74304c03a883c3ef4a7259d95c1300bbb60
Nov 01 05:45:42.295113 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.295076    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vzzng" event={"ID":"70c203e1-de50-4a9e-a77f-1c7a97335ba1","Type":"ContainerStarted","Data":"11e82adff7c925facf0db21edd8d03cdbd2e11f58dc1aa9cac049ae47e19a284"}
Nov 01 05:45:42.297808 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.297776    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzwxh" event={"ID":"dc4f5277-87b1-43e4-a47e-2a42cc73f400","Type":"ContainerStarted","Data":"f1fdff628e7c106a2c9aabd0805b5c9d4fdbedeacc6522e263c60907c9054982"}
Nov 01 05:45:42.297929 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.297863    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzwxh" event={"ID":"dc4f5277-87b1-43e4-a47e-2a42cc73f400","Type":"ContainerStarted","Data":"fa3635c9725c8992ad54c36ff4dec74304c03a883c3ef4a7259d95c1300bbb60"}
Nov 01 05:45:42.299348 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.299300    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wbwlt" event={"ID":"7bec7b62-a68f-4579-8bcc-14eb1c26f163","Type":"ContainerStarted","Data":"7cd95c36d197031d96e0c501a4ca42b34bc5cfc02e3a99bd4a16572278a13990"}
Nov 01 05:45:42.640347 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.640311    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"]
Nov 01 05:45:42.642845 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.642799    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.695210 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.694968    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 01 05:45:42.695210 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.695028    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 01 05:45:42.695210 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.695102    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 05:45:42.695210 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.695115    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 05:45:42.695210 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.695177    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7x6xl\""
Nov 01 05:45:42.695558 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.695240    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 05:45:42.700420 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.700395    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"]
Nov 01 05:45:42.810546 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.810453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.810698 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.810600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.810698 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.810655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2sm\" (UniqueName: \"kubernetes.io/projected/6b81cb01-44c1-4d62-8f59-443a7f27be7d-kube-api-access-2m2sm\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.810698 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.810695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b81cb01-44c1-4d62-8f59-443a7f27be7d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.911868 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.911813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2sm\" (UniqueName: \"kubernetes.io/projected/6b81cb01-44c1-4d62-8f59-443a7f27be7d-kube-api-access-2m2sm\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.912029 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.911883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b81cb01-44c1-4d62-8f59-443a7f27be7d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.912029 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.911943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.912029 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.911998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.913008 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.912985    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b81cb01-44c1-4d62-8f59-443a7f27be7d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.915246 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.914806    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.915246 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.915198    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:42.973977 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:42.973940    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2sm\" (UniqueName: \"kubernetes.io/projected/6b81cb01-44c1-4d62-8f59-443a7f27be7d-kube-api-access-2m2sm\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:43.255876 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.255619    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:45:43.303939 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.303906    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzwxh" event={"ID":"dc4f5277-87b1-43e4-a47e-2a42cc73f400","Type":"ContainerStarted","Data":"22a587f33559eea0a52e78c40ee9e90dcf8e33968f3d4d6f11939a349c4d0601"}
Nov 01 05:45:43.494604 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.494575    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c87d46cd7-qjbsf"]
Nov 01 05:45:43.496675 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.496651    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.549001 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.548974    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 05:45:43.549100 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.549072    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 05:45:43.549100 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.549086    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 05:45:43.549218 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.549113    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 05:45:43.549218 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.549130    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 05:45:43.549218 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.549168    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mqtzq\""
Nov 01 05:45:43.549218 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.549177    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 05:45:43.549218 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.549211    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 05:45:43.556816 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.556795    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c87d46cd7-qjbsf"]
Nov 01 05:45:43.618225 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.618175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.618225 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.618202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6m62\" (UniqueName: \"kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.618370 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.618242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.618370 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.618275    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.618370 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.618299    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.618370 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.618337    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.618370 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.618362    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.675678 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.675647    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"]
Nov 01 05:45:43.681787 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:43.680676    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b81cb01_44c1_4d62_8f59_443a7f27be7d.slice/crio-87121fb94f85ab70a732b9f2d2e9331309f451e350a98e3be5453cfc88d38b97 WatchSource:0}: Error finding container 87121fb94f85ab70a732b9f2d2e9331309f451e350a98e3be5453cfc88d38b97: Status 404 returned error can't find the container with id 87121fb94f85ab70a732b9f2d2e9331309f451e350a98e3be5453cfc88d38b97
Nov 01 05:45:43.718728 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.718691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.718861 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.718749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6m62\" (UniqueName: \"kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.718861 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.718821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.718991 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.718889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.718991 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.718933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.718991 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.718975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.719132 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.719001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.719182 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.719144    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.723589 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.723534    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.723771 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.723748    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.723771 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.723763    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.727342 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.727315    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.732368 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.732345    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.779881 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.779794    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6m62\" (UniqueName: \"kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:43.808559 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:43.808515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:44.189324 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:44.189257    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c87d46cd7-qjbsf"]
Nov 01 05:45:44.192169 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:44.192142    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5088be44_c26e_4b66_b3f0_11b5815d711a.slice/crio-673cb71b8aca04aebcab5e38e392dda45a84e0732bc8c1781de7c83b90a61d28 WatchSource:0}: Error finding container 673cb71b8aca04aebcab5e38e392dda45a84e0732bc8c1781de7c83b90a61d28: Status 404 returned error can't find the container with id 673cb71b8aca04aebcab5e38e392dda45a84e0732bc8c1781de7c83b90a61d28
Nov 01 05:45:44.309907 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:44.309868    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vzzng" event={"ID":"70c203e1-de50-4a9e-a77f-1c7a97335ba1","Type":"ContainerStarted","Data":"d561ba3766afcf2e564a16280cd2b6ade538aee1f907e8f618635c5b7c9229dc"}
Nov 01 05:45:44.310375 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:44.309936    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vzzng" event={"ID":"70c203e1-de50-4a9e-a77f-1c7a97335ba1","Type":"ContainerStarted","Data":"ca130f597e555e1f3ff9bfffa6ef4826e1fb4ef93e6976ffc9544b5324eda031"}
Nov 01 05:45:44.310375 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:44.309992    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:44.311505 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:44.311477    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c87d46cd7-qjbsf" event={"ID":"5088be44-c26e-4b66-b3f0-11b5815d711a","Type":"ContainerStarted","Data":"673cb71b8aca04aebcab5e38e392dda45a84e0732bc8c1781de7c83b90a61d28"}
Nov 01 05:45:44.312925 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:44.312893    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h" event={"ID":"6b81cb01-44c1-4d62-8f59-443a7f27be7d","Type":"ContainerStarted","Data":"87121fb94f85ab70a732b9f2d2e9331309f451e350a98e3be5453cfc88d38b97"}
Nov 01 05:45:44.427782 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:44.427702    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vzzng" podStartSLOduration=2.189149137 podStartE2EDuration="4.427685229s" totalImagesPullingTime="2.238536092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:44.427118497 +0000 UTC m=+52.823899893" watchObservedRunningTime="2026-11-01 05:45:44.427685229 +0000 UTC m=+52.824466626"
Nov 01 05:45:45.318664 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:45.318626    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzwxh" event={"ID":"dc4f5277-87b1-43e4-a47e-2a42cc73f400","Type":"ContainerStarted","Data":"0760c526c2ac083d3787cd11c2ff737836b3665124e1367d589ec57d2784565b"}
Nov 01 05:45:45.321964 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:45.321935    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wbwlt" event={"ID":"7bec7b62-a68f-4579-8bcc-14eb1c26f163","Type":"ContainerStarted","Data":"170fbb744d551806e847d9c44e7a6d094a28fa225646e350d4d505c70f74d5b9"}
Nov 01 05:45:45.436911 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:45.436852    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pzwxh" podStartSLOduration=2.3324509239999998 podStartE2EDuration="5.436811553s" totalImagesPullingTime="3.104360629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:45:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:45.436251696 +0000 UTC m=+53.833033120" watchObservedRunningTime="2026-11-01 05:45:45.436811553 +0000 UTC m=+53.833592954"
Nov 01 05:45:45.552504 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:45.552441    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wbwlt" podStartSLOduration=2.289820137 podStartE2EDuration="5.552422629s" totalImagesPullingTime="3.262602492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:45.550857255 +0000 UTC m=+53.947638673" watchObservedRunningTime="2026-11-01 05:45:45.552422629 +0000 UTC m=+53.949204025"
Nov 01 05:45:46.326935 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:46.326839    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h" event={"ID":"6b81cb01-44c1-4d62-8f59-443a7f27be7d","Type":"ContainerStarted","Data":"8f69c59bfae3ec29d65bb86bc8464f5e0bea168e959235d7ae75cac5382050f1"}
Nov 01 05:45:46.326935 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:46.326901    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h" event={"ID":"6b81cb01-44c1-4d62-8f59-443a7f27be7d","Type":"ContainerStarted","Data":"347181e8496944dae5d6617a3978903ce08df3a312497bd72a7eba10467dd13a"}
Nov 01 05:45:46.455114 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:46.455065    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h" podStartSLOduration=2.685118398 podStartE2EDuration="4.455049993s" totalImagesPullingTime="1.769931595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:46.454645458 +0000 UTC m=+54.851426880" watchObservedRunningTime="2026-11-01 05:45:46.455049993 +0000 UTC m=+54.851831388"
Nov 01 05:45:48.252880 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:48.252853    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:45:48.337091 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:48.337038    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c87d46cd7-qjbsf" event={"ID":"5088be44-c26e-4b66-b3f0-11b5815d711a","Type":"ContainerStarted","Data":"021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a"}
Nov 01 05:45:48.621695 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:48.621587    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c87d46cd7-qjbsf" podStartSLOduration=2.434769619 podStartE2EDuration="5.621573128s" totalImagesPullingTime="3.186803509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:48.620953944 +0000 UTC m=+57.017735359" watchObservedRunningTime="2026-11-01 05:45:48.621573128 +0000 UTC m=+57.018354578"
Nov 01 05:45:49.085566 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.085526    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"]
Nov 01 05:45:49.087486 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.087472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.140839 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.140793    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 05:45:49.140839 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.140793    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 05:45:49.141023 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.140906    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dmhjl\""
Nov 01 05:45:49.147348 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.147322    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"]
Nov 01 05:45:49.194389 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.194360    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"]
Nov 01 05:45:49.196224 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.196209    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.248676 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.248647    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 05:45:49.248814 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.248798    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 05:45:49.248886 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.248860    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 05:45:49.248936 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.248923    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-k6p64\""
Nov 01 05:45:49.258865 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.258816    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"]
Nov 01 05:45:49.263041 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.263013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmqv\" (UniqueName: \"kubernetes.io/projected/a13ea0c7-7fe8-4aec-805c-26197f386653-kube-api-access-7kmqv\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.263164 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.263063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.263238 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.263172    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13ea0c7-7fe8-4aec-805c-26197f386653-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.263300 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.263235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.281438 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.281410    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w7sbw"]
Nov 01 05:45:49.283544 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.283513    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.348533 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.348471    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f427j\""
Nov 01 05:45:49.348642 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.348530    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 05:45:49.348642 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.348550    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 05:45:49.348713 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.348655    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 05:45:49.364082 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs94g\" (UniqueName: \"kubernetes.io/projected/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-api-access-cs94g\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.364132 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.364132 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.364198 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c92dc00-2bba-4345-a8c3-b36c72a24056-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.364269 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.364302 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmqv\" (UniqueName: \"kubernetes.io/projected/a13ea0c7-7fe8-4aec-805c-26197f386653-kube-api-access-7kmqv\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.364333 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.364333 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13ea0c7-7fe8-4aec-805c-26197f386653-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.364422 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364339    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.364422 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364367    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.364969 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.364950    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13ea0c7-7fe8-4aec-805c-26197f386653-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.366759 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.366740    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.366821 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.366744    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.424622 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.424591    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmqv\" (UniqueName: \"kubernetes.io/projected/a13ea0c7-7fe8-4aec-805c-26197f386653-kube-api-access-7kmqv\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.465129 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-textfile\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.465229 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-metrics-client-ca\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.465276 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.465392 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465377    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.465470 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.465511 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-wtmp\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.465544 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.465671 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs94g\" (UniqueName: \"kubernetes.io/projected/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-api-access-cs94g\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.465671 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-root\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.465783 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.465783 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-sys\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.466072 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-tls\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.466072 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.466072 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465890    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c92dc00-2bba-4345-a8c3-b36c72a24056-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.466072 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.465920    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvph\" (UniqueName: \"kubernetes.io/projected/c39c479a-5489-41fe-b2e4-28f033f0e021-kube-api-access-zdvph\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.466272 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.466243    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c92dc00-2bba-4345-a8c3-b36c72a24056-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.466272 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.466246    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.466413 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.466394    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.467586 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.467566    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.468226 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.468210    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.526005 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.525976    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs94g\" (UniqueName: \"kubernetes.io/projected/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-api-access-cs94g\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.566987 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.566965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-wtmp\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567058 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.566992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567058 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-root\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567118 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-sys\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567118 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-tls\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567200 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvph\" (UniqueName: \"kubernetes.io/projected/c39c479a-5489-41fe-b2e4-28f033f0e021-kube-api-access-zdvph\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567200 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567148    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-wtmp\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567200 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-textfile\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567200 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567159    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-root\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567200 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567184    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-sys\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567409 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-metrics-client-ca\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567409 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567503 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567431    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-textfile\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.567708 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567693    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-metrics-client-ca\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.568016 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.567994    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.569261 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.569241    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.569380 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.569365    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-tls\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.627357 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.627310    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvph\" (UniqueName: \"kubernetes.io/projected/c39c479a-5489-41fe-b2e4-28f033f0e021-kube-api-access-zdvph\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.698706 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.698676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:45:49.807912 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.807874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:45:49.894526 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:49.894459    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:45:49.903790 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:49.903765    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39c479a_5489_41fe_b2e4_28f033f0e021.slice/crio-d9966bd8d64a294bba93cb46eb1f9a140b96d95a853a4305b00f7bd67ea89a1e WatchSource:0}: Error finding container d9966bd8d64a294bba93cb46eb1f9a140b96d95a853a4305b00f7bd67ea89a1e: Status 404 returned error can't find the container with id d9966bd8d64a294bba93cb46eb1f9a140b96d95a853a4305b00f7bd67ea89a1e
Nov 01 05:45:50.116644 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:50.116609    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"]
Nov 01 05:45:50.162019 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:50.161986    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13ea0c7_7fe8_4aec_805c_26197f386653.slice/crio-1bdac70f97a707f2eed75589c07afa87f0b696c939936de12f3ba512d569ee9b WatchSource:0}: Error finding container 1bdac70f97a707f2eed75589c07afa87f0b696c939936de12f3ba512d569ee9b: Status 404 returned error can't find the container with id 1bdac70f97a707f2eed75589c07afa87f0b696c939936de12f3ba512d569ee9b
Nov 01 05:45:50.172751 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:50.172725    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"]
Nov 01 05:45:50.176958 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:50.176928    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c92dc00_2bba_4345_a8c3_b36c72a24056.slice/crio-fac50f2c5eb9a93d494bde85c39583047b8327129e9a519cf82352546c9a1919 WatchSource:0}: Error finding container fac50f2c5eb9a93d494bde85c39583047b8327129e9a519cf82352546c9a1919: Status 404 returned error can't find the container with id fac50f2c5eb9a93d494bde85c39583047b8327129e9a519cf82352546c9a1919
Nov 01 05:45:50.344723 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:50.344688    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7sbw" event={"ID":"c39c479a-5489-41fe-b2e4-28f033f0e021","Type":"ContainerStarted","Data":"d9966bd8d64a294bba93cb46eb1f9a140b96d95a853a4305b00f7bd67ea89a1e"}
Nov 01 05:45:50.345973 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:50.345951    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m" event={"ID":"8c92dc00-2bba-4345-a8c3-b36c72a24056","Type":"ContainerStarted","Data":"fac50f2c5eb9a93d494bde85c39583047b8327129e9a519cf82352546c9a1919"}
Nov 01 05:45:50.347740 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:50.347718    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78" event={"ID":"a13ea0c7-7fe8-4aec-805c-26197f386653","Type":"ContainerStarted","Data":"1c4a6d17e6c4d245e6e123cb21cb465aeeb6f61a363b852f54e328d8c20ed239"}
Nov 01 05:45:50.347819 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:50.347758    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78" event={"ID":"a13ea0c7-7fe8-4aec-805c-26197f386653","Type":"ContainerStarted","Data":"4b3d4ac6028d5194fce980945ceb2ed4a8fe8ee298cbb4d3442dda2b89a3ec3d"}
Nov 01 05:45:50.347819 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:50.347777    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78" event={"ID":"a13ea0c7-7fe8-4aec-805c-26197f386653","Type":"ContainerStarted","Data":"1bdac70f97a707f2eed75589c07afa87f0b696c939936de12f3ba512d569ee9b"}
Nov 01 05:45:51.925203 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:51.925168    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-584b5d845f-c29z8"]
Nov 01 05:45:51.928346 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:51.928296    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:51.980396 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:51.980366    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 05:45:51.980396 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:51.980376    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a35uihkrv29ol\""
Nov 01 05:45:51.980547 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:51.980379    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 05:45:51.980547 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:51.980451    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 05:45:51.990735 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:51.990713    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-584b5d845f-c29z8"]
Nov 01 05:45:52.080524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.080502    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 05:45:52.080524 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.080509    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p56vd\""
Nov 01 05:45:52.080650 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.080573    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088223    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-tmp\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-metrics-client-ca\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-grpc-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088391    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.089801 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.088457    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2vz\" (UniqueName: \"kubernetes.io/projected/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-kube-api-access-df2vz\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189291 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189216    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-metrics-client-ca\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189291 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189477 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189477 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189419    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-grpc-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189477 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189627 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2vz\" (UniqueName: \"kubernetes.io/projected/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-kube-api-access-df2vz\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189627 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189727 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.189727 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.189677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-tmp\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.190398 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.190340    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-metrics-client-ca\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.190398 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.190366    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-tmp\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.192152 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.192113    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.192152 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.192122    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.192681 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.192658    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.192730 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.192699    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-grpc-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.192898 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.192872    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.193101 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.193082    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.252487 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.252451    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2vz\" (UniqueName: \"kubernetes.io/projected/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-kube-api-access-df2vz\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.356735 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.356703    2575 generic.go:356] "Generic (PLEG): container finished" podID="c39c479a-5489-41fe-b2e4-28f033f0e021" containerID="4fd1f599a4d544bb608190953a17c0bf11fc55d10ad4ce117c73524c430e2d8b" exitCode=0
Nov 01 05:45:52.356937 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.356782    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7sbw" event={"ID":"c39c479a-5489-41fe-b2e4-28f033f0e021","Type":"ContainerDied","Data":"4fd1f599a4d544bb608190953a17c0bf11fc55d10ad4ce117c73524c430e2d8b"}
Nov 01 05:45:52.359174 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.359140    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m" event={"ID":"8c92dc00-2bba-4345-a8c3-b36c72a24056","Type":"ContainerStarted","Data":"85b1a08a4e1dbf04afd8d4f60ead0f0c7bb1efb94d308750c60098d67bac83d7"}
Nov 01 05:45:52.359267 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.359191    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m" event={"ID":"8c92dc00-2bba-4345-a8c3-b36c72a24056","Type":"ContainerStarted","Data":"02dc4d76f95f22c1f1197ce19734ac181ce2d1fe23c5c71e950c93301c46e71e"}
Nov 01 05:45:52.359267 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.359224    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m" event={"ID":"8c92dc00-2bba-4345-a8c3-b36c72a24056","Type":"ContainerStarted","Data":"a620260fb7ff2c0d1c478736d0219abe0fb7dbd0469be46edcc3403a0605b1d0"}
Nov 01 05:45:52.361400 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.361379    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78" event={"ID":"a13ea0c7-7fe8-4aec-805c-26197f386653","Type":"ContainerStarted","Data":"4474e919e4de469771b429d2cc8a36118f73a8c8da1d9aa4607499eedfbb70fc"}
Nov 01 05:45:52.550290 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.550249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:52.593479 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.593418    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m" podStartSLOduration=1.8317395109999999 podStartE2EDuration="3.593405495s" totalImagesPullingTime="1.761665984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:52.592030285 +0000 UTC m=+60.988811702" watchObservedRunningTime="2026-11-01 05:45:52.593405495 +0000 UTC m=+60.990186891"
Nov 01 05:45:52.709178 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.709122    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78" podStartSLOduration=2.28924531 podStartE2EDuration="3.709108722s" totalImagesPullingTime="1.419863412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:52.708398249 +0000 UTC m=+61.105179692" watchObservedRunningTime="2026-11-01 05:45:52.709108722 +0000 UTC m=+61.105890118"
Nov 01 05:45:52.980767 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:52.980741    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-584b5d845f-c29z8"]
Nov 01 05:45:52.983087 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:52.983055    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e9bca5_9a75_4cbf_a3b8_7b64ad4f41f8.slice/crio-890ac0fbfd484509616d042f1a01963e8f15f1508bc032fc762f2e93a701e362 WatchSource:0}: Error finding container 890ac0fbfd484509616d042f1a01963e8f15f1508bc032fc762f2e93a701e362: Status 404 returned error can't find the container with id 890ac0fbfd484509616d042f1a01963e8f15f1508bc032fc762f2e93a701e362
Nov 01 05:45:53.366193 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:53.366108    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7sbw" event={"ID":"c39c479a-5489-41fe-b2e4-28f033f0e021","Type":"ContainerStarted","Data":"0e9750182916faccef1131535f59b613d8e337971a82a4aaf1b65123e860ba05"}
Nov 01 05:45:53.366193 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:53.366173    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7sbw" event={"ID":"c39c479a-5489-41fe-b2e4-28f033f0e021","Type":"ContainerStarted","Data":"01f85ad93e6346dab60f25d4d909db7b3ca4529fee55dd56d7ce5fa37a3fae80"}
Nov 01 05:45:53.367312 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:53.367290    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8" event={"ID":"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8","Type":"ContainerStarted","Data":"890ac0fbfd484509616d042f1a01963e8f15f1508bc032fc762f2e93a701e362"}
Nov 01 05:45:53.485255 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:53.485199    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w7sbw" podStartSLOduration=2.99574781 podStartE2EDuration="4.48518515s" totalImagesPullingTime="1.405084751s" totalInitContainerRuntime="84.352589ms" isStatefulPod=true podCreationTimestamp="2026-11-01 05:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:53.48372339 +0000 UTC m=+61.880504788" watchObservedRunningTime="2026-11-01 05:45:53.48518515 +0000 UTC m=+61.881966572"
Nov 01 05:45:53.808621 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:53.808590    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:53.808952 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:53.808930    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:53.813159 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:53.813139    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:54.323731 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.323708    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vzzng"
Nov 01 05:45:54.376075 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.376048    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:45:54.491377 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.491349    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-54986ff55f-nlwd8"]
Nov 01 05:45:54.494809 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.494787    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.514178 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.514153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8fs\" (UniqueName: \"kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.514290 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.514187    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.514353 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.514314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.514682 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.514647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.514819 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.514730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.514819 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.514789    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.514819 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.514807    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.515010 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.514847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.549335 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.549297    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54986ff55f-nlwd8"]
Nov 01 05:45:54.615901 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.615814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.615901 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.615888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.616109 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.615933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.616109 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.615973    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.616109 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.615998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.616109 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.616021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.616109 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.616083    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8fs\" (UniqueName: \"kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.616350 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.616109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.616653 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.616623    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.617313 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.617291    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.617455 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.617436    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.617938 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.617909    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.619017 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.618996    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.620056 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.620038    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.639889 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.639854    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 05:45:54.648120 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.648082    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.678148 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.678123    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8fs\" (UniqueName: \"kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:54.809218 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:54.809188    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:45:55.188272 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:55.188188    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54986ff55f-nlwd8"]
Nov 01 05:45:55.191460 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:55.191434    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ab10be_8c8b_41a0_a6b9_21d43eb30c94.slice/crio-4285f8b10928230dbe192e9c23c19a9e01de42951db6a7cc277c975a6d024711 WatchSource:0}: Error finding container 4285f8b10928230dbe192e9c23c19a9e01de42951db6a7cc277c975a6d024711: Status 404 returned error can't find the container with id 4285f8b10928230dbe192e9c23c19a9e01de42951db6a7cc277c975a6d024711
Nov 01 05:45:55.376971 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:55.376910    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54986ff55f-nlwd8" event={"ID":"28ab10be-8c8b-41a0-a6b9-21d43eb30c94","Type":"ContainerStarted","Data":"cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5"}
Nov 01 05:45:55.377430 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:55.376984    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54986ff55f-nlwd8" event={"ID":"28ab10be-8c8b-41a0-a6b9-21d43eb30c94","Type":"ContainerStarted","Data":"4285f8b10928230dbe192e9c23c19a9e01de42951db6a7cc277c975a6d024711"}
Nov 01 05:45:55.494123 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:55.494080    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-54986ff55f-nlwd8" podStartSLOduration=1.494065265 podStartE2EDuration="1.494065265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:55.493295391 +0000 UTC m=+63.890076806" watchObservedRunningTime="2026-11-01 05:45:55.494065265 +0000 UTC m=+63.890846657"
Nov 01 05:45:56.382081 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:56.382048    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8" event={"ID":"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8","Type":"ContainerStarted","Data":"4709db17aea3d0b4d7ae5eb5ab663f6888a043b694e6c2a6d5476dfc4ea8022b"}
Nov 01 05:45:56.382478 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:56.382117    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8" event={"ID":"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8","Type":"ContainerStarted","Data":"310e95814218bd22bddaf488f94f9043ba96603b0331880e6abba7c014a79ef6"}
Nov 01 05:45:56.382478 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:56.382143    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8" event={"ID":"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8","Type":"ContainerStarted","Data":"6a5c5eeb5d622390e441b81e51440464426a0370c0dcf5a9720e1b46879ede2b"}
Nov 01 05:45:57.040536 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.040506    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:57.040626 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.040547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:57.091783 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.091752    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:45:57.103608 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.103551    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:57.127473 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.127410    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:45:57.133294 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.133271    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:57.142158 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.141798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:57.192885 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.192858    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 05:45:57.253758 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.253731    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 05:45:57.265433 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.265403    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:57.314181 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.314155    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87mzr\""
Nov 01 05:45:57.318121 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.318106    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v8m4f\""
Nov 01 05:45:57.323855 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.323821    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:45:57.327754 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.327741    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:45:57.400813 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.400785    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8" event={"ID":"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8","Type":"ContainerStarted","Data":"4c38eb4ea8b9e56cfb2b5f9ee58bcb6cd356a89b08bb35fb909162c53b323fe4"}
Nov 01 05:45:57.400813 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.400863    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8" event={"ID":"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8","Type":"ContainerStarted","Data":"2e1eb45b2d74ac4bbef392ace7bf4561d739acef08d0f5b4d51cb73578f4d3ab"}
Nov 01 05:45:57.400813 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.400889    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8" event={"ID":"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8","Type":"ContainerStarted","Data":"bfa6077e5f506473cf5a941c55164b38565a511b32044750dba7f2008ef39cff"}
Nov 01 05:45:57.401410 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.400958    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:45:57.524432 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.524379    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8" podStartSLOduration=2.814853808 podStartE2EDuration="6.524360808s" totalImagesPullingTime="3.709507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:57.523177175 +0000 UTC m=+65.919958596" watchObservedRunningTime="2026-11-01 05:45:57.524360808 +0000 UTC m=+65.921142207"
Nov 01 05:45:57.611700 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.610440    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4d6z\""
Nov 01 05:45:57.617887 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.617860    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:45:57.700114 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.700069    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5qzs"]
Nov 01 05:45:57.702609 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:57.702583    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf4eb38_7ce4_45fe_8c38_0d8907a8861f.slice/crio-80a2acccd22521d02bfa713737a9aa13581fe2c2088098d08e8406600afbf56f WatchSource:0}: Error finding container 80a2acccd22521d02bfa713737a9aa13581fe2c2088098d08e8406600afbf56f: Status 404 returned error can't find the container with id 80a2acccd22521d02bfa713737a9aa13581fe2c2088098d08e8406600afbf56f
Nov 01 05:45:57.749389 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:57.749370    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hh9pk"]
Nov 01 05:45:57.751600 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:57.751571    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639fd403_7cfd_438d_90cb_07c383b61678.slice/crio-d1b6f1f5b2958ef7c320380d034c8678d72b202a6ad64e745cae0cdcdacfb5c8 WatchSource:0}: Error finding container d1b6f1f5b2958ef7c320380d034c8678d72b202a6ad64e745cae0cdcdacfb5c8: Status 404 returned error can't find the container with id d1b6f1f5b2958ef7c320380d034c8678d72b202a6ad64e745cae0cdcdacfb5c8
Nov 01 05:45:58.006772 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:58.006713    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gj86"]
Nov 01 05:45:58.009384 ip-10-0-138-190 kubenswrapper[2575]: W1101 05:45:58.009354    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ef7b0f_d31d_41bf_a91e_06c64350ec49.slice/crio-9f6b54b728a880e437fc9d9c22697911185de8ebb21ec08ce484824d4f9d2baa WatchSource:0}: Error finding container 9f6b54b728a880e437fc9d9c22697911185de8ebb21ec08ce484824d4f9d2baa: Status 404 returned error can't find the container with id 9f6b54b728a880e437fc9d9c22697911185de8ebb21ec08ce484824d4f9d2baa
Nov 01 05:45:58.405796 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:58.405716    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hh9pk" event={"ID":"639fd403-7cfd-438d-90cb-07c383b61678","Type":"ContainerStarted","Data":"d1b6f1f5b2958ef7c320380d034c8678d72b202a6ad64e745cae0cdcdacfb5c8"}
Nov 01 05:45:58.406881 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:58.406857    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gj86" event={"ID":"e6ef7b0f-d31d-41bf-a91e-06c64350ec49","Type":"ContainerStarted","Data":"9f6b54b728a880e437fc9d9c22697911185de8ebb21ec08ce484824d4f9d2baa"}
Nov 01 05:45:58.407950 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:45:58.407918    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r5qzs" event={"ID":"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f","Type":"ContainerStarted","Data":"80a2acccd22521d02bfa713737a9aa13581fe2c2088098d08e8406600afbf56f"}
Nov 01 05:46:00.419710 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:46:00.419671    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hh9pk" event={"ID":"639fd403-7cfd-438d-90cb-07c383b61678","Type":"ContainerStarted","Data":"32f77bd1f8d9bc2359936bb931e3d5cd012955d28fa98bb30b7389351e46bf34"}
Nov 01 05:46:00.420253 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:46:00.419735    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hh9pk" event={"ID":"639fd403-7cfd-438d-90cb-07c383b61678","Type":"ContainerStarted","Data":"571b47ad56896372d55ca7a6e2ed1076b52ed884086c984f4523e1294e7b395f"}
Nov 01 05:46:00.535934 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:46:00.535874    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hh9pk" podStartSLOduration=66.953322911 podStartE2EDuration="1m8.53585296s" totalImagesPullingTime="1.582530049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:00.535330394 +0000 UTC m=+68.932111787" watchObservedRunningTime="2026-11-01 05:46:00.53585296 +0000 UTC m=+68.932634351"
Nov 01 05:46:02.430090 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:46:02.430054    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gj86" event={"ID":"e6ef7b0f-d31d-41bf-a91e-06c64350ec49","Type":"ContainerStarted","Data":"70e4908b2231f070f2da4054f2b6819fcb618f156b01797a37b32b0f6387d10e"}
Nov 01 05:46:02.430562 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:46:02.430161    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:46:02.546003 ip-10-0-138-190 kubenswrapper[2575]: I1101 05:46:02.545941    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4gj86" podStartSLOduration=66.879622228 podStartE2EDuration="1m10.545924057s" totalImagesPullingTime="3.666301829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:02.544215843 +0000 UTC m=+70.940997239" watchObservedRunningTime="2026-11-01 05:46:02.545924057 +0000 UTC m=+70.942705651"
Nov 01 05:46:03.053069 ip-10-0-138-190 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 05:46:03.065246 ip-10-0-138-190 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 05:46:03.065415 ip-10-0-138-190 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 05:46:03.065462 ip-10-0-138-190 systemd[1]: kubelet.service: Consumed 4.062s CPU time.
Nov 01 05:46:03.086646 ip-10-0-138-190 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 05:46:03.150485 ip-10-0-138-190 kubenswrapper[5510]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:46:03.150485 ip-10-0-138-190 kubenswrapper[5510]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 05:46:03.150485 ip-10-0-138-190 kubenswrapper[5510]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:46:03.150485 ip-10-0-138-190 kubenswrapper[5510]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:46:03.152469 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152392    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152471    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152475    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152479    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152482    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152484    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152487    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152490    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152493    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152495    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152498    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152501    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152503    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152506    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152508    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152511    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152513    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152517    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:03.152511 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152519    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152522    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152525    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152528    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152530    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152533    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152535    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152541    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152544    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152546    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152549    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152551    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152554    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152556    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152559    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152561    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152564    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152566    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152568    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152571    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:03.152988 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152573    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152575    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152578    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152580    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152583    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152585    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152588    5510 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152590    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152593    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152595    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152598    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152601    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152604    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152607    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152610    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152612    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152615    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152618    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152620    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:03.153473 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152623    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152625    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152628    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152630    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152633    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152635    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152638    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152640    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152643    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152645    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152648    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152650    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152653    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152655    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152658    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152660    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152662    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152665    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152667    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152671    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:03.153969 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152673    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152676    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152679    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152681    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152683    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152686    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152688    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152691    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152693    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152696    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152698    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152700    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152703    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152705    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152708    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152710    5510 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152714    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152718    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152722    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:03.154485 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152724    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152727    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152729    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152732    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152734    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152737    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152739    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152743    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152745    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.152748    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153525    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153532    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153535    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153539    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153543    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153546    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153548    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153551    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153554    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153557    5510 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:03.155001 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153560    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153562    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153565    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153567    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153569    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153572    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153574    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153577    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153579    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153582    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153584    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153586    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153598    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153601    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153605    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153607    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153610    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153613    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153615    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:03.155510 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153618    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153620    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153623    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153626    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153629    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153631    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153634    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153638    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153641    5510 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153643    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153646    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153648    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153651    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153653    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153656    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153658    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153660    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153663    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153665    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153668    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:03.155985 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153670    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153673    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153675    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153677    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153680    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153682    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153686    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153691    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153694    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153697    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153701    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153703    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153707    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153709    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153712    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153714    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153717    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153720    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:03.156462 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153723    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153727    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153730    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153733    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153736    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153738    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153741    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153743    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153746    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153749    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153751    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153754    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153756    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153759    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153774    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153777    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153781    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153784    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153787    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153789    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:03.156958 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153792    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153794    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153797    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153799    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153802    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153805    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153807    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153809    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153812    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153814    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153816    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153819    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153821    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153840    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153842    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153845    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153848    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153851    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.153855    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154330    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:03.157439 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154335    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154338    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154341    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154343    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154346    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154348    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154351    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154354    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154356    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154359    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154361    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154364    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154366    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154368    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154371    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154373    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154376    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154378    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154381    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154384    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:03.157948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154386    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154388    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154391    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154394    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154396    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154399    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154401    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154404    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154408    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154412    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154414    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154417    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154420    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154422    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154425    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154427    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154430    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154432    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154435    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:03.158442 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154437    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154440    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154442    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154445    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154447    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154450    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154452    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154455    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154457    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154462    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154465    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154467    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154470    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154472    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154475    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154477    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154480    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154482    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154485    5510 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:03.158927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154487    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154490    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154493    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154496    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154500    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154502    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154505    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154507    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154510    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154512    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154514    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154517    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154519    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154522    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154524    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154526    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154529    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154533    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154536    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154538    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:03.159398 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154540    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154543    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154546    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154549    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154552    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154554    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154556    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154559    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154561    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154564    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154566    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154569    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154571    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154574    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154576    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154579    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154581    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154584    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154586    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:03.159887 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154589    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154593    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154596    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154600    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154603    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154606    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154608    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.154611    5510 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154699    5510 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154706    5510 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154718    5510 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154724    5510 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154731    5510 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154735    5510 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154742    5510 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154748    5510 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154751    5510 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154754    5510 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154757    5510 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154760    5510 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154763    5510 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154766    5510 flags.go:64] FLAG: --cgroup-root=""
Nov 01 05:46:03.160380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154769    5510 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154772    5510 flags.go:64] FLAG: --client-ca-file=""
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154776    5510 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154779    5510 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154783    5510 flags.go:64] FLAG: --cluster-domain=""
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154785    5510 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154788    5510 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154792    5510 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154795    5510 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154799    5510 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154802    5510 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154805    5510 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154808    5510 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154811    5510 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154813    5510 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154816    5510 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154819    5510 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154822    5510 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154843    5510 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154846    5510 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154851    5510 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154853    5510 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154856    5510 flags.go:64] FLAG: --enable-server="true"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154859    5510 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154864    5510 flags.go:64] FLAG: --event-burst="100"
Nov 01 05:46:03.160960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154869    5510 flags.go:64] FLAG: --event-qps="50"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154872    5510 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154875    5510 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154878    5510 flags.go:64] FLAG: --eviction-hard=""
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154882    5510 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154885    5510 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154888    5510 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154891    5510 flags.go:64] FLAG: --eviction-soft=""
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154894    5510 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154897    5510 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154899    5510 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154902    5510 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154905    5510 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154908    5510 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154911    5510 flags.go:64] FLAG: --feature-gates=""
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154915    5510 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154918    5510 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154922    5510 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154925    5510 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154927    5510 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154930    5510 flags.go:64] FLAG: --help="false"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154933    5510 flags.go:64] FLAG: --hostname-override="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154936    5510 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154939    5510 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 05:46:03.161563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154942    5510 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154945    5510 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154948    5510 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154952    5510 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154955    5510 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154958    5510 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154960    5510 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154963    5510 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154967    5510 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154970    5510 flags.go:64] FLAG: --kube-reserved=""
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154973    5510 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154976    5510 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154979    5510 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154982    5510 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154985    5510 flags.go:64] FLAG: --lock-file=""
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154987    5510 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154990    5510 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154993    5510 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.154999    5510 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155001    5510 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155004    5510 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155007    5510 flags.go:64] FLAG: --logging-format="text"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155010    5510 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155014    5510 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 05:46:03.162216 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155017    5510 flags.go:64] FLAG: --manifest-url=""
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155020    5510 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155024    5510 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155027    5510 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155031    5510 flags.go:64] FLAG: --max-pods="110"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155034    5510 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155037    5510 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155040    5510 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155043    5510 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155045    5510 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155048    5510 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155053    5510 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155060    5510 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155063    5510 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155066    5510 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155069    5510 flags.go:64] FLAG: --pod-cidr=""
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155073    5510 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155075    5510 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155078    5510 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155081    5510 flags.go:64] FLAG: --port="10250"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155084    5510 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155087    5510 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-04599b72d757a00b1"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155090    5510 flags.go:64] FLAG: --qos-reserved=""
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155093    5510 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155096    5510 flags.go:64] FLAG: --register-node="true"
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155098    5510 flags.go:64] FLAG: --register-with-taints=""
Nov 01 05:46:03.162805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155102    5510 flags.go:64] FLAG: --registry-burst="10"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155105    5510 flags.go:64] FLAG: --registry-qps="5"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155107    5510 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155110    5510 flags.go:64] FLAG: --reserved-memory=""
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155114    5510 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155117    5510 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155119    5510 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155123    5510 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155125    5510 flags.go:64] FLAG: --runonce="false"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155128    5510 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155131    5510 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155134    5510 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155137    5510 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155139    5510 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155142    5510 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155145    5510 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155148    5510 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155153    5510 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155155    5510 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155158    5510 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155161    5510 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155164    5510 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155168    5510 flags.go:64] FLAG: --system-cgroups=""
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155171    5510 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155176    5510 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 05:46:03.163568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155179    5510 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155182    5510 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155186    5510 flags.go:64] FLAG: --tls-min-version=""
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155189    5510 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155191    5510 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155194    5510 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155197    5510 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155200    5510 flags.go:64] FLAG: --v="2"
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155204    5510 flags.go:64] FLAG: --version="false"
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155208    5510 flags.go:64] FLAG: --vmodule=""
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155212    5510 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155215    5510 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155361    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155365    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155369    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155373    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155378    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155381    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155384    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155387    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155389    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155392    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:03.164567 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155394    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155397    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155400    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155403    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155405    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155408    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155410    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155414    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155417    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155419    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155422    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155424    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155427    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155429    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155431    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155434    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155436    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155439    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155441    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155444    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:03.165118 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155446    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155449    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155451    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155454    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155456    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155459    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155462    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155464    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155467    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155469    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155472    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155474    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155477    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155479    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155483    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155485    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155488    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155491    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155494    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155497    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:03.165677 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155500    5510 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155502    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155505    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155507    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155510    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155512    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155515    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155517    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155520    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155523    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155525    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155528    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155531    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155533    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155535    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155538    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155540    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155543    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155545    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:03.166180 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155551    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155553    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155556    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155559    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155561    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155563    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155566    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155569    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155572    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155574    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155576    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155580    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155584    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155587    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155590    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155592    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155595    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155597    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155600    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:03.166695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155602    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155604    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155607    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155609    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155611    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155614    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155616    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155619    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155621    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155623    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155626    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155628    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155631    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155633    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155636    5510 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155639    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155641    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:03.167220 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155644    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.155669    5510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155706    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155712    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155715    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155718    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155721    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155723    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155726    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155729    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155732    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155734    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155736    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155739    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155742    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155744    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:03.167723 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155747    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155750    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155753    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155756    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155759    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155761    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155764    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155766    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155769    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155771    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155774    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155776    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155779    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155781    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155784    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155786    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155789    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155791    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155794    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:03.168176 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155798    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155800    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155803    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155805    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155808    5510 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155810    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155812    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155815    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155817    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155820    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155836    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155840    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155844    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155847    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155849    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155852    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155855    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155857    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155860    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:03.168670 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155862    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155865    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155867    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155870    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155872    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155875    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155878    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155880    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155883    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155886    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155888    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155891    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155893    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155897    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155900    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155902    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155905    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155907    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155910    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155912    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:03.169300 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155915    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155917    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155919    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155922    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155924    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155927    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155931    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155933    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155936    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155938    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155941    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155943    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155945    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155948    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155950    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155953    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155958    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155961    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155963    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155966    5510 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:03.169937 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155968    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155971    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155975    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155977    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155980    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155984    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155987    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155989    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155992    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155995    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.155997    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.156001    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.156004    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.156006    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.160139    5510 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.160152    5510 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.160169    5510 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.160174    5510 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160240    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:03.170422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160245    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160248    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160251    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160253    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160256    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160259    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160261    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160264    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160266    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160270    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160273    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160276    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160278    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160280    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160283    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160291    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160294    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160297    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160300    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160302    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:03.170903 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160305    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160307    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160310    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160312    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160315    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160317    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160319    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160321    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160324    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160326    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160329    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160331    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160333    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160336    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160338    5510 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160340    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160343    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160345    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160348    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160350    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:03.171380 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160353    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160356    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160358    5510 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160360    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160363    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160365    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160367    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160370    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160378    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160381    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160384    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160387    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160389    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160392    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160394    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160397    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160400    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160402    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160404    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:03.171948 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160407    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160409    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160413    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160416    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160419    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160421    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160423    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160426    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160429    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160431    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160433    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160436    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160438    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160441    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160444    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160446    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160449    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160451    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160453    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:03.172401 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160456    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160458    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160461    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160469    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160471    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160474    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160476    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160479    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160481    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160483    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160486    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160488    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160490    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160493    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160495    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160498    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160500    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160502    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160505    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:03.172871 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160509    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160513    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160516    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160519    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160522    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160524    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160527    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160529    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.160555    5510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160779    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160785    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160788    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160790    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160794    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160798    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160801    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:03.173320 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160804    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160812    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160815    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160817    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160819    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160822    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160843    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160846    5510 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160849    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160854    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160857    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160859    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160862    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160864    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160867    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160869    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160873    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160875    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160878    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:03.173708 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160881    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160883    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160886    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160888    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160891    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160893    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160896    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160898    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160900    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160904    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160907    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160910    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160913    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160915    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160917    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160927    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160930    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160932    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160935    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160937    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:03.174218 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160939    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160942    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160946    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160948    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160950    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160953    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160955    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160958    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160960    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160963    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160965    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160968    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160971    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160973    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160976    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160979    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160981    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160984    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160992    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160994    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:03.174695 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.160997    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161000    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161002    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161005    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161008    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161010    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161013    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161015    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161023    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161026    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161028    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161031    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161033    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161035    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161039    5510 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161042    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161044    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161046    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161049    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:03.175202 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161051    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161054    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161056    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161058    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161061    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161063    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161065    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161068    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161070    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161072    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161075    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161077    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161080    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161082    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161084    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161087    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161089    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161092    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161094    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161097    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:03.175652 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161099    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.161124    5510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161172    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161176    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161178    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161181    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161184    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161186    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161189    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161191    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161193    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161196    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161198    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161201    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161203    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161206    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:03.176139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161209    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161212    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161214    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161217    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161219    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161222    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161226    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161229    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161233    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161235    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161238    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161240    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161243    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161245    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161248    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161250    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161252    5510 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161255    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161257    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:03.176527 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161265    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161268    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161271    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161273    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161276    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161278    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161281    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161283    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161285    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161288    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161290    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161293    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161295    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161299    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161301    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161304    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161306    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161309    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161312    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:03.176994 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161315    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161318    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161321    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161323    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161325    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161328    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161330    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161333    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161335    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161337    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161340    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161342    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161344    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161347    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161357    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161360    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161362    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161364    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161367    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161369    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:03.177596 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161371    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161374    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161376    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161379    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161381    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161383    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161388    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161390    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161393    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161395    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161398    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161400    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161403    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161406    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161409    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161412    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161414    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161417    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161419    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161422    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:03.178113 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161424    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161427    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161429    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161431    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161435    5510 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161437    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161440    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161448    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161450    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161453    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161455    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161458    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161460    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:03.161463    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.161507    5510 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.162472    5510 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.162527    5510 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.162723    5510 server.go:1014] "Starting client certificate rotation"
Nov 01 05:46:03.178583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.163326    5510 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:46:03.179267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.163492    5510 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 05:39:52 +0000 UTC" deadline="2028-06-28 09:06:25.575726601 +0000 UTC"
Nov 01 05:46:03.179267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.163536    5510 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14523h20m22.412193985s"
Nov 01 05:46:03.179267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.164637    5510 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:46:03.179267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.165769    5510 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:46:03.179267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.166897    5510 log.go:25] "Validated CRI v1 runtime API"
Nov 01 05:46:03.179267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.168622    5510 log.go:25] "Validated CRI v1 image API"
Nov 01 05:46:03.179267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.169394    5510 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 05:46:03.179267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.171472    5510 fs.go:135] Filesystem UUIDs: map[79bc8aad-df98-42fa-845b-5444c4e26efa:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9a8b236c-6760-4f6b-a627-33f1c418f808:/dev/nvme0n1p4]
Nov 01 05:46:03.179516 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.171489    5510 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11e82adff7c925facf0db21edd8d03cdbd2e11f58dc1aa9cac049ae47e19a284/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11e82adff7c925facf0db21edd8d03cdbd2e11f58dc1aa9cac049ae47e19a284/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bdac70f97a707f2eed75589c07afa87f0b696c939936de12f3ba512d569ee9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bdac70f97a707f2eed75589c07afa87f0b696c939936de12f3ba512d569ee9b/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d303c776eb7055a0791497d4ffc65a788f07a980e5273a10704e1d90d273243/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d303c776eb7055a0791497d4ffc65a788f07a980e5273a10704e1d90d273243/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/378300804cf7bf6f8296483feb9326b50e13fb9ce944d1d4cf49317be9ffc314/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/378300804cf7bf6f8296483feb9326b50e13fb9ce944d1d4cf49317be9ffc314/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ab6e90da6d95c8112eaeda37c694f31a99222794269476058f8f2bbe1294031/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ab6e90da6d95c8112eaeda37c694f31a99222794269476058f8f2bbe1294031/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4285f8b10928230dbe192e9c23c19a9e01de42951db6a7cc277c975a6d024711/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4285f8b10928230dbe192e9c23c19a9e01de42951db6a7cc277c975a6d024711/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6136a52e6139f67c300dd3518fe7f9abca2f66b15db5003dfdc39054036638c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6136a52e6139f67c300dd3518fe7f9abca2f66b15db5003dfdc39054036638c8/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6540a00891483588ac51739dc2b762ae97b83227d69bd9cf538a1b950193d17b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6540a00891483588ac51739dc2b762ae97b83227d69bd9cf538a1b950193d17b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/673cb71b8aca04aebcab5e38e392dda45a84e0732bc8c1781de7c83b90a61d28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/673cb71b8aca04aebcab5e38e392dda45a84e0732bc8c1781de7c83b90a61d28/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e272943af250680ae24b73a8ade0f01216b9f3666177b1f23e5a3f7ef70ad6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e272943af250680ae24b73a8ade0f01216b9f3666177b1f23e5a3f7ef70ad6f/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cd95c36d197031d96e0c501a4ca42b34bc5cfc02e3a99bd4a16572278a13990/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cd95c36d197031d96e0c501a4ca42b34bc5cfc02e3a99bd4a16572278a13990/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80a2acccd22521d02bfa713737a9aa13581fe2c2088098d08e8406600afbf56f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80a2acccd22521d02bfa713737a9aa13581fe2c2088098d08e8406600afbf56f/userdata/shm major:0 minor:404 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87121fb94f85ab70a732b9f2d2e9331309f451e350a98e3be5453cfc88d38b97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87121fb94f85ab70a732b9f2d2e9331309f451e350a98e3be5453cfc88d38b97/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/890ac0fbfd484509616d042f1a01963e8f15f1508bc032fc762f2e93a701e362/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/890ac0fbfd484509616d042f1a01963e8f15f1508bc032fc762f2e93a701e362/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c0a22b4970447cc1a5f07f5fdbdd042131dcc9592ca0459b50b47223e97894e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c0a22b4970447cc1a5f07f5fdbdd042131dcc9592ca0459b50b47223e97894e/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fdc637a494aa8ee0816ca41c7c57c4aa15fe98004de388c0bcd81710cabd828/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fdc637a494aa8ee0816ca41c7c57c4aa15fe98004de388c0bcd81710cabd828/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99bc589564b993e0e97b22eeabd99d2220bd7b1de7c6fcce8e0bed144c380b7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99bc589564b993e0e97b22eeabd99d2220bd7b1de7c6fcce8e0bed144c380b7e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f6b54b728a880e437fc9d9c22697911185de8ebb21ec08ce484824d4f9d2baa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f6b54b728a880e437fc9d9c22697911185de8ebb21ec08ce484824d4f9d2baa/userdata/shm major:0 minor:406 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1b6f1f5b2958ef7c320380d034c8678d72b202a6ad64e745cae0cdcdacfb5c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1b6f1f5b2958ef7c320380d034c8678d72b202a6ad64e745cae0cdcdacfb5c8/userdata/shm major:0 minor:402 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9966bd8d64a294bba93cb46eb1f9a140b96d95a853a4305b00f7bd67ea89a1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9966bd8d64a294bba93cb46eb1f9a140b96d95a853a4305b00f7bd67ea89a1e/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dccf9082c6e01ffa488c5481609e9b4c605e685a43664984bea03d90d7e755de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dccf9082c6e01ffa488c5481609e9b4c605e685a43664984bea03d90d7e755de/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa3635c9725c8992ad54c36ff4dec74304c03a883c3ef4a7259d95c1300bbb60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa3635c9725c8992ad54c36ff4dec74304c03a883c3ef4a7259d95c1300bbb60/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fac50f2c5eb9a93d494bde85c39583047b8327129e9a519cf82352546c9a1919/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fac50f2c5eb9a93d494bde85c39583047b8327129e9a519cf82352546c9a1919/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0593c2e4-1a34-445b-8d91-35a1b0888afe/volumes/kubernetes.io~projected/kube-api-access-gr5hf:{mountpoint:/var/lib/kubelet/pods/0593c2e4-1a34-445b-8d91-35a1b0888afe/volumes/kubernetes.io~projected/kube-api-access-gr5hf major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~projected/kube-api-access-df2vz:{mountpoint:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~projected/kube-api-access-df2vz major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11dab409-45c8-49c5-9300-24db5bfa7a9f/volumes/kubernetes.io~projected/kube-api-access-tskrb:{mountpoint:/var/lib/kubelet/pods/11dab409-45c8-49c5-9300-24db5bfa7a9f/volumes/kubernetes.io~projected/kube-api-access-tskrb major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volumes/kubernetes.io~projected/kube-api-access-d8xlr:{mountpoint:/var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volumes/kubernetes.io~projected/kube-api-access-d8xlr major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~projected/kube-api-access-5z8fs:{mountpoint:/var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~projected/kube-api-access-5z8fs major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c402d6b-1936-4cea-a20b-42e4a2ec19a9/volumes/kubernetes.io~projected/kube-api-access-vs7kx:{mountpoint:/var/lib/kubelet/pods/2c402d6b-1936-4cea-a20b-42e4a2ec19a9/volumes/kubernetes.io~projected/kube-api-access-vs7kx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b530358-dea0-471a-85d7-73749d6fa3af/volumes/kubernetes.io~projected/kube-api-access-jrcdv:{mountpoint:/var/lib/kubelet/pods/3b530358-dea0-471a-85d7-73749d6fa3af/volumes/kubernetes.io~projected/kube-api-access-jrcdv major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~projected/kube-api-access-p6m62:{mountpoint:/var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~projected/kube-api-access-p6m62 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/639fd403-7cfd-438d-90cb-07c383b61678/volumes/kubernetes.io~projected/kube-api-access-k8fww:{mountpoint:/var/lib/kubelet/pods/639fd403-7cfd-438d-90cb-07c383b61678/volumes/kubernetes.io~projected/kube-api-access-k8fww major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/639fd403-7cfd-438d-90cb-07c383b61678/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/639fd403-7cfd-438d-90cb-07c383b61678/volumes/kubernetes.io~secret/metrics-certs major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/667231b8-a7ce-45ce-adc5-61b1e84c5f12/volumes/kubernetes.io~projected/kube-api-access-ct72m:{mountpoint:/var/lib/kubelet/pods/667231b8-a7ce-45ce-adc5-61b1e84c5f12/volumes/kubernetes.io~projected/kube-api-access-ct72m major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~projected/kube-api-access-2m2sm:{mountpoint:/var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~projected/kube-api-access-2m2sm major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~projected/kube-api-access-gq5lj:{mountpoint:/var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~projected/kube-api-access-gq5lj major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70c203e1-de50-4a9e-a77f-1c7a97335ba1/volumes/kubernetes.io~projected/kube-api-access-qmpkf:{mountpoint:/var/lib/kubelet/pods/70c203e1-de50-4a9e-a77f-1c7a97335ba1/volumes/kubernetes.io~projected/kube-api-access-qmpkf major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70c203e1-de50-4a9e-a77f-1c7a97335ba1/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/70c203e1-de50-4a9e-a77f-1c7a97335ba1/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76bf1b5b-01e1-4f37-b0a9-67a64366ecca/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/76bf1b5b-01e1-4f37-b0a9-67a64366ecca/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bec7b62-a68f-4579-8bcc-14eb1c26f163/volumes/kubernetes.io~projected/kube-api-access-lpz5k:{mountpoint:/var/lib/kubelet/pods/7bec7b62-a68f-4579-8bcc-14eb1c26f163/volumes/kubernetes.io~projected/kube-api-access-lpz5k major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bec7b62-a68f-4579-8bcc-14eb1c26f163/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7bec7b62-a68f-4579-8bcc-14eb1c26f163/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~projected/kube-api-access-cs94g:{mountpoint:/var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~projected/kube-api-access-cs94g major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~projected/kube-api-access-7kmqv:{mountpoint:/var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~projected/kube-api-access-7kmqv major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~projected/kube-api-access-zdvph:{mountpoint:/var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~projected/kube-api-access-zdvph major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc4f5277-87b1-43e4-a47e-2a42cc73f400/volumes/kubernetes.io~projected/kube-api-access-pcdfb:{mountpoint:/var/lib/kubelet/pods/dc4f5277-87b1-43e4-a47e-2a42cc73f400/volumes/kubernetes.io~projected/kube-api-access-pcdfb major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc4f5277-87b1-43e4-a47e-2a42cc73f400/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/dc4f5277-87b1-43e4-a47e-2a42cc73f400/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6ef7b0f-d31d-41bf-a91e-06c64350ec49/volumes/kubernetes.io~projected/kube-api-access-bwm65:{mountpoint:/var/lib/kubelet/pods/e6ef7b0f-d31d-41bf-a91e-06c64350ec49/volumes/kubernetes.io~projected/kube-api-access-bwm65 major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e72e536d-221c-43a0-9a31-0062010750ed/volumes/kubernetes.io~projected/kube-api-access-2qg9l:{mountpoint:/var/lib/kubelet/pods/e72e536d-221c-43a0-9a31-0062010750ed/volumes/kubernetes.io~projected/kube-api-access-2qg9l major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/604b835ad78f9658a56089df43b51a6f3416d6d00e0349bf1d264a776f01e086/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0c60188e8207fe1abde69b716f0c0bd788adbce3a79bfb630d458448266f267f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9526e7f8d2697fc9762ce10db6f407fc480b4abac068af03870c72651a5889ab/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/890675c5baaebc32ab29c4d33f4e80ea0143c8e54e128527506a3663ad5deac1/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c344d10245eb086fccda32d659da00341ef87fcd71cf4dc225874536136536dd/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0cddf98efd033e2fe0a76cdf408584f43b3fc636adc2d089d59e9bf5a5896443/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/260a8b8fa66004af10c4d64f00f485451d84ed5824475ddaaa35ca4e6638cd56/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/dea1d9e5f9701978590cef23002538160effed051dab1cc78bbcd81f2fd3f3e4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/561a7f2ad5ede08e187184f201738ea425d005b7474040f8cad4de94cab8fe30/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/964b30a05c58f72ddc1281dd8ca61cf8c1d70053d24f0f277cffeca8296f81a1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/719f6c7cf4f126b534ea98b3c0b4ca58716006ffa7390db4dfadb019467b2510/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/724c99c630a2770c207fbd1099287242e43b437cd7df7a065e1cc67112867a96/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f58c01857358365d94527cb0864258bfa99b11c864982759a84bfd568ec4bd12/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a9c0d2a82d7bfdf9d835b07d5f4998a9d8e34ec59fc4d0caaae121148cfe03d2/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/112ead992f9ed78db01ca87dfc74d4c2108c9efc11035a76eec3a4affadc1fd6/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/28f3d020ee478ee89be49620b1a5bf1448232a680d6295463eb7d8fcb6861825/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/edd899ea5d82b2b77986c5672e0255f9e368c3a81061b370debaa7f04fb6c37c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4791d6a26f6d38124e8c4e1c24c93e48b05f655bfae7e1ce8386aa6b27df314c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e971eb9403261bdcdc9e7dc2bee6b2736846de99092dff7467db0f7b34104594/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2234040b33692c2894a420fbd41587d4e2696bb334289916a0d699879d5f6cb7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ae21447ee269af03459214f7cd8ad1ce417c7f87169781878063226e8efd0a12/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/cc26003bc92aaeed21b555e006e1947997104ef64cc7b85a32662636ab82f357/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2b0846042717e270329455fc8b5bef06c67119f60aea4985ffb4f106c31cc9dd/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/fc842a1b3c7fa435ba7d8ea63d3e50cbdd46d4cd1b082d4dad9e2400412d37cf/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/c276666aa59277401f40d79889a591a6db834184499619546d7d8a80eeba7a9b/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/c25cabcac1a7b787070d90f80d246dd3e025ad57af4899ca3170bdb018fed6c3/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/bb6203cca376a4b06f4db6ca9e8fe2f4c2b60a1b91174f92d0c668b14700686c/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/3b0158632dca122da88c5cf1b3519b607cf8a9ae05433cad3f9fb9843ab3817b/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/3609c7f4e765c216ccc52121506386d2f5221e11c0022e2248f1efa1ad669a86/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/762a9602816a6e1715fe1f59b75c54aa5c34538acbca0fe94d9bbd3b78bee84d/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/87c08d9261ad5f088321ba734f39fc5b2693e60da2276e9abfaf3ead7e2d9dc9/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/6aff9bc959de77605e4957bd31de2f276e27fb80b286ac1c14d2132b9e954724/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/0d811430752300b1cc22ff7490e2cca5b976bc3ae89d224a7d6bce44db24a3da/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/b1a3ac99d9745b94d727c5a339b08194e5202213189faecabd18e78e352a9c36/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/f463c6fbf08b17b7a1a68338037db339ae046fb7a2794cf31c0c26a897f946f4/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/58a654c2cec17789a6d2278f2520c9a0f4df0c603a3aca7dd868746fda8952eb/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/b13203c1cbb3c6eef8f8a48a95e30a83268165f2193e27e8e2fe0bbcecdb2686/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/4e678437f7629e9341181ab0b83d17b702d2ae00039c0b80876b94b3bf1f6b00/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/b2e7e9d6ba4710673eb5dd1b7c5d99e3ec8d0e23c5a45fa5d85ff59388878307/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/49d11ca228d7921338df588b360b88124b2ca80cc0f521043d292152cd8ca072/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a40b2856cb152998062c0ab2208340e19ed662fc9249cdb4e94d0d1661d0b4d5/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/1a475cd5f9fef56e0325b8d91569477113d9981caaf0ea0570f51a65905f7acf/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/8cd00c834f64cf79752553ba3186875348580f84211995e3a944c1e9d0312022/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/8678f5d140149dab73df3b4e0da8374e1e50951a343302bbe42b7125590f222e/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/a8843f16e21d76d01954958bb1c4ffad4fef31dd813af10e6974f1f6909b40de/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/7078a9ef4b714fc62981366bdc3a78a8275d020c3f3e26a7b4b7f56d519edeb8/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/750827c868f09775e2c98405fea3890ee4b099c182407a45aa510174e41fe637/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/bda7a56b0f3cc8c21f9a4597fc607574a0b18bb6e0c14f2326f855801fd22cc7/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/9216021a6da81471b5c602ec386a7f49607bfb77b92f6d94766e5e9a0fdec322/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/fa4d038f943c917e320fd6a150491c58ef4abc8f201a36aaea70a154449b861e/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/cd81c9e8a1b8c9b0d69c7f4788b326cb07942318de5a63c8d2f44bdb7dde4643/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/af07251c4f192b328fb054bb9150c2e0d05383923ea1786614c4b6904f7b91fa/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/7e1d9a4f46d03b9b2d7b5033065aba603ff7c66f8d9ad2c61d4a7e97159e802f/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/822aacef84951c4056575177a59459d144bbb116b5a8e65e482950b585d71775/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/45faf5cdacaf718728c3d24b1cb2ebe754d38300075b19ae306441c249d684de/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/dc202127283b88dfe09e702d9e1899904bba9af3959ed2c97c0a8964badcb4c8/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/3e750ba8ee179671c6fcca51e6b2e9b55a9127654325aca0ee761977880ae17f/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/8cf84da7ecbef1db0c6aea5ffa7741098258208708c25d84400edfd1816d96d0/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/3a0d4872d5879a29d659090cf8d44e4669bf6fe6029fb7c81cf4d4e6899a2d06/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/e5f455e5bfc26c62fe0b765f6beb1a9935b615d1fb388aaaeef133bd1cb6860a/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/cfbb290ba7eda965169f8fb58950e9ea3364d02aea7fa84debbfd9be0366bc60/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/86ecd407f048ad8b748e2c09642365e4dfbef6995cae1c3787a4ec6affda6912/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/21923e76ed0bfbf2cf4f2032dafb271ed8e142b24807bfecaf3f1e6c6fdf45d8/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/44b150a50d5f72c515143ae18527788886206e7e8e41b6a2ec2e0638c56a66ca/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/20ccc762b853e5694f3a91b770a74bcdddd4aa726ddbec54db00c4df9d903c7e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/89369e2e980074d2f4c9368690f599e995ff8a64887185ac81e46993a78ab545/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/4990b29b131fcfb1da2dec34161b222f2acdc869898f580c12b1f15af989e574/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/e7487a3cd0229e8535bd6074adbee2402d19b1241c2b506359bb51780f64cdb3/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/34d99f55864d771cc1f23a25e6f4e32f77f3a9c0501e8d9b61f41915640c1761/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/7e858c8aec09db9c4e37b20711cb4a826dbba7ea43a7d37809802082800da107/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/14858bcaeb35017ece6110d3861caaf5f4249a46cdf3cf0f234ef7b2191b33b8/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/976c2f0cc951e7205d2156ca52c13c7b07956f301f900ee6a09aa0b9a8cda4e0/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 05:46:03.179797 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.178641    5510 manager.go:217] Machine: {Timestamp:2026-11-01 05:46:03.178199372 +0000 UTC m=+0.056172773 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3128657 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f2d0739f46b3983ffebe90f2e274c SystemUUID:ec2f2d07-39f4-6b39-83ff-ebe90f2e274c BootID:eba5dcef-5e2f-44e6-bb67-0f61318124ba Filesystems:[{Device:/var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d303c776eb7055a0791497d4ffc65a788f07a980e5273a10704e1d90d273243/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bdac70f97a707f2eed75589c07afa87f0b696c939936de12f3ba512d569ee9b/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/378300804cf7bf6f8296483feb9326b50e13fb9ce944d1d4cf49317be9ffc314/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:400 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7bec7b62-a68f-4579-8bcc-14eb1c26f163/volumes/kubernetes.io~projected/kube-api-access-lpz5k DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~projected/kube-api-access-gq5lj DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4285f8b10928230dbe192e9c23c19a9e01de42951db6a7cc277c975a6d024711/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f6b54b728a880e437fc9d9c22697911185de8ebb21ec08ce484824d4f9d2baa/userdata/shm DeviceMajor:0 DeviceMinor:406 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/667231b8-a7ce-45ce-adc5-61b1e84c5f12/volumes/kubernetes.io~projected/kube-api-access-ct72m DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cd95c36d197031d96e0c501a4ca42b34bc5cfc02e3a99bd4a16572278a13990/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80a2acccd22521d02bfa713737a9aa13581fe2c2088098d08e8406600afbf56f/userdata/shm DeviceMajor:0 DeviceMinor:404 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11dab409-45c8-49c5-9300-24db5bfa7a9f/volumes/kubernetes.io~projected/kube-api-access-tskrb DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99bc589564b993e0e97b22eeabd99d2220bd7b1de7c6fcce8e0bed144c380b7e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~projected/kube-api-access-2m2sm DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~projected/kube-api-access-cs94g DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87121fb94f85ab70a732b9f2d2e9331309f451e350a98e3be5453cfc88d38b97/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/639fd403-7cfd-438d-90cb-07c383b61678/volumes/kubernetes.io~projected/kube-api-access-k8fww DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6136a52e6139f67c300dd3518fe7f9abca2f66b15db5003dfdc39054036638c8/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b530358-dea0-471a-85d7-73749d6fa3af/volumes/kubernetes.io~projected/kube-api-access-jrcdv DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6540a00891483588ac51739dc2b762ae97b83227d69bd9cf538a1b950193d17b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc4f5277-87b1-43e4-a47e-2a42cc73f400/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b81cb01-44c1-4d62-8f59-443a7f27be7d/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:355 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ee449a4-59d0-40a5-af37-297952887cc3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dc4f5277-87b1-43e4-a47e-2a42cc73f400/volumes/kubernetes.io~projected/kube-api-access-pcdfb DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa3635c9725c8992ad54c36ff4dec74304c03a883c3ef4a7259d95c1300bbb60/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1b6f1f5b2958ef7c320380d034c8678d72b202a6ad64e745cae0cdcdacfb5c8/userdata/shm DeviceMajor:0 DeviceMinor:402 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2c402d6b-1936-4cea-a20b-42e4a2ec19a9/volumes/kubernetes.io~projected/kube-api-access-vs7kx DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70c203e1-de50-4a9e-a77f-1c7a97335ba1/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9966bd8d64a294bba93cb46eb1f9a140b96d95a853a4305b00f7bd67ea89a1e/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e72e536d-221c-43a0-9a31-0062010750ed/volumes/kubernetes.io~projected/kube-api-access-2qg9l DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~projected/kube-api-access-zdvph DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/890ac0fbfd484509616d042f1a01963e8f15f1508bc032fc762f2e93a701e362/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:247 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0593c2e4-1a34-445b-8d91-35a1b0888afe/volumes/kubernetes.io~projected/kube-api-access-gr5hf DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16f463e7-7075-4621-95ae-60c2fefdb950/volumes/kubernetes.io~projected/kube-api-access-d8xlr DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ab6e90da6d95c8112eaeda37c694f31a99222794269476058f8f2bbe1294031/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e6ef7b0f-d31d-41bf-a91e-06c64350ec49/volumes/kubernetes.io~projected/kube-api-access-bwm65 DeviceMajor:0 DeviceMinor:401 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a13ea0c7-7fe8-4aec-805c-26197f386653/volumes/kubernetes.io~projected/kube-api-access-7kmqv DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:356 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e272943af250680ae24b73a8ade0f01216b9f3666177b1f23e5a3f7ef70ad6f/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7bec7b62-a68f-4579-8bcc-14eb1c26f163/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes/kubernetes.io~projected/kube-api-access-p6m62 DeviceMajor:0 DeviceMinor:248 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/673cb71b8aca04aebcab5e38e392dda45a84e0732bc8c1781de7c83b90a61d28/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c39c479a-5489-41fe-b2e4-28f033f0e021/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/76bf1b5b-01e1-4f37-b0a9-67a64366ecca/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes/kubernetes.io~projected/kube-api-access-5z8fs DeviceMajor:0 DeviceMinor:357 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11e82adff7c925facf0db21edd8d03cdbd2e11f58dc1aa9cac049ae47e19a284/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fdc637a494aa8ee0816ca41c7c57c4aa15fe98004de388c0bcd81710cabd828/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70c203e1-de50-4a9e-a77f-1c7a97335ba1/volumes/kubernetes.io~projected/kube-api-access-qmpkf DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/volumes/kubernetes.io~projected/kube-api-access-df2vz DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c0a22b4970447cc1a5f07f5fdbdd042131dcc9592ca0459b50b47223e97894e/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8c92dc00-2bba-4345-a8c3-b36c72a24056/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fac50f2c5eb9a93d494bde85c39583047b8327129e9a519cf82352546c9a1919/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/639fd403-7cfd-438d-90cb-07c383b61678/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:396 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dccf9082c6e01ffa488c5481609e9b4c605e685a43664984bea03d90d7e755de/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:11e82adff7c925f MacAddress:ba:d1:36:04:0f:14 Speed:10000 Mtu:8901} {Name:1bdac70f97a707f MacAddress:2a:b0:74:b7:78:81 Speed:10000 Mtu:8901} {Name:4285f8b10928230 MacAddress:c6:6b:0c:56:37:03 Speed:10000 Mtu:8901} {Name:673cb71b8aca04a MacAddress:3e:72:92:9b:d1:9f Speed:10000 Mtu:8901} {Name:7cd95c36d197031 MacAddress:ce:99:e0:4e:e9:8a Speed:10000 Mtu:8901} {Name:80a2acccd22521d MacAddress:3e:29:87:42:63:08 Speed:10000 Mtu:8901} {Name:87121fb94f85ab7 MacAddress:e6:d3:aa:53:30:31 Speed:10000 Mtu:8901} {Name:890ac0fbfd48450 MacAddress:8a:82:a2:84:ee:2f Speed:10000 Mtu:8901} {Name:9f6b54b728a880e MacAddress:e6:a4:5f:20:8e:f9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:4f:22:74:26:db Speed:0 Mtu:9001} {Name:br-int MacAddress:c6:ad:b1:3d:e1:31 Speed:0 Mtu:8901} {Name:d1b6f1f5b2958ef MacAddress:c6:f1:eb:9c:a7:e5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:4f:22:74:26:db Speed:0 Mtu:9001} {Name:fa3635c9725c899 MacAddress:1a:64:97:1d:a3:b8 Speed:10000 Mtu:8901} {Name:fac50f2c5eb9a93 MacAddress:e2:a5:34:4b:12:2d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:76:19:e5:95:4e:46 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:52:35:90:02:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 05:46:03.179797 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.179033    5510 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 05:46:03.179797 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.179075    5510 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 05:46:03.179797 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.179336    5510 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 05:46:03.180242 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.179360    5510 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-190.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 05:46:03.180242 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180022    5510 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 05:46:03.180242 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180037    5510 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 05:46:03.180242 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180056    5510 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:46:03.180242 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180077    5510 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:46:03.180242 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180180    5510 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 05:46:03.180523 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180450    5510 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 05:46:03.180612 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180598    5510 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 05:46:03.180662 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180646    5510 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 05:46:03.180770 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180747    5510 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 05:46:03.180894 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180780    5510 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 05:46:03.180894 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180798    5510 kubelet.go:405] "Adding apiserver pod source"
Nov 01 05:46:03.180894 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.180807    5510 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 05:46:03.182331 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.182313    5510 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 05:46:03.182331 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.182328    5510 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:46:03.182786 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.182770    5510 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 05:46:03.185272 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185258    5510 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 05:46:03.185272 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185272    5510 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:46:03.185372 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185363    5510 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 05:46:03.185561 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185550    5510 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 05:46:03.185601 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185580    5510 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 05:46:03.185683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185673    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 05:46:03.185718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185693    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 05:46:03.185718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185704    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 05:46:03.185718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185716    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 05:46:03.185793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185729    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 05:46:03.185793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185742    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 05:46:03.185793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185757    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 05:46:03.185793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185770    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 05:46:03.185793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185786    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 05:46:03.185989 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185800    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 05:46:03.185989 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185819    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 05:46:03.185989 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185845    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 05:46:03.185989 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185871    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 05:46:03.185989 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.185892    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 05:46:03.189341 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.189328    5510 server.go:1263] "Started kubelet"
Nov 01 05:46:03.189531 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.189425    5510 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 05:46:03.189635 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.189503    5510 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 05:46:03.189635 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.189598    5510 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 05:46:03.189939 ip-10-0-138-190 systemd[1]: Started Kubernetes Kubelet.
Nov 01 05:46:03.190720 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.190377    5510 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 05:46:03.191090 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.191056    5510 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 05:46:03.196217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.196041    5510 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 05:46:03.196217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.196068    5510 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 05:46:03.196217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.196131    5510 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 05:46:03.196217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.196143    5510 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 05:46:03.196217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.196131    5510 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:53 +0000 UTC" deadline="2028-04-25 18:32:29.340994598 +0000 UTC"
Nov 01 05:46:03.196217 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.196147    5510 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-190.us-west-1.compute.internal\" not found"
Nov 01 05:46:03.196217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.196168    5510 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12996h46m26.14483318s"
Nov 01 05:46:03.196575 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.196238    5510 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 05:46:03.198088 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198066    5510 factory.go:153] Registering CRI-O factory
Nov 01 05:46:03.198088 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198085    5510 factory.go:223] Registration of the crio container factory successfully
Nov 01 05:46:03.198234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198138    5510 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 05:46:03.198234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198147    5510 factory.go:55] Registering systemd factory
Nov 01 05:46:03.198234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198154    5510 factory.go:223] Registration of the systemd container factory successfully
Nov 01 05:46:03.198234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198169    5510 factory.go:103] Registering Raw factory
Nov 01 05:46:03.198234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198181    5510 manager.go:1199] Started watching for new ooms in manager
Nov 01 05:46:03.198508 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198496    5510 manager.go:319] Starting recovery of all containers
Nov 01 05:46:03.198870 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198785    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/configmap/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:46:03.198943 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198875    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" volumeName="kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca" seLinuxMountContext=""
Nov 01 05:46:03.198943 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198893    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b81cb01-44c1-4d62-8f59-443a7f27be7d" volumeName="kubernetes.io/configmap/6b81cb01-44c1-4d62-8f59-443a7f27be7d-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:46:03.198943 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198905    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c92dc00-2bba-4345-a8c3-b36c72a24056" volumeName="kubernetes.io/empty-dir/8c92dc00-2bba-4345-a8c3-b36c72a24056-volume-directive-shadow" seLinuxMountContext=""
Nov 01 05:46:03.198943 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198918    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f463e7-7075-4621-95ae-60c2fefdb950" volumeName="kubernetes.io/secret/16f463e7-7075-4621-95ae-60c2fefdb950-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 05:46:03.198943 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198930    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" volumeName="kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config" seLinuxMountContext=""
Nov 01 05:46:03.198943 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198943    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" volumeName="kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198955    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11dab409-45c8-49c5-9300-24db5bfa7a9f" volumeName="kubernetes.io/projected/11dab409-45c8-49c5-9300-24db5bfa7a9f-kube-api-access-tskrb" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198970    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f463e7-7075-4621-95ae-60c2fefdb950" volumeName="kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-config" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198984    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" volumeName="kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.198996    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5088be44-c26e-4b66-b3f0-11b5815d711a" volumeName="kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199009    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b81cb01-44c1-4d62-8f59-443a7f27be7d" volumeName="kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199020    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c479a-5489-41fe-b2e4-28f033f0e021" volumeName="kubernetes.io/projected/c39c479a-5489-41fe-b2e4-28f033f0e021-kube-api-access-zdvph" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199031    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee449a4-59d0-40a5-af37-297952887cc3" volumeName="kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-tmp" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199044    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c203e1-de50-4a9e-a77f-1c7a97335ba1" volumeName="kubernetes.io/projected/70c203e1-de50-4a9e-a77f-1c7a97335ba1-kube-api-access-qmpkf" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199069    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0593c2e4-1a34-445b-8d91-35a1b0888afe" volumeName="kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-daemon-config" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199084    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/projected/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-kube-api-access-df2vz" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199097    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b530358-dea0-471a-85d7-73749d6fa3af" volumeName="kubernetes.io/projected/3b530358-dea0-471a-85d7-73749d6fa3af-kube-api-access-jrcdv" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199109    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bec7b62-a68f-4579-8bcc-14eb1c26f163" volumeName="kubernetes.io/projected/7bec7b62-a68f-4579-8bcc-14eb1c26f163-kube-api-access-lpz5k" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199120    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c479a-5489-41fe-b2e4-28f033f0e021" volumeName="kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199132    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc4f5277-87b1-43e4-a47e-2a42cc73f400" volumeName="kubernetes.io/secret/dc4f5277-87b1-43e4-a47e-2a42cc73f400-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199144    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199156    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199177    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c402d6b-1936-4cea-a20b-42e4a2ec19a9" volumeName="kubernetes.io/configmap/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-iptables-alerter-script" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199189    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5088be44-c26e-4b66-b3f0-11b5815d711a" volumeName="kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert" seLinuxMountContext=""
Nov 01 05:46:03.199199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199201    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b81cb01-44c1-4d62-8f59-443a7f27be7d" volumeName="kubernetes.io/projected/6b81cb01-44c1-4d62-8f59-443a7f27be7d-kube-api-access-2m2sm" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199214    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76bf1b5b-01e1-4f37-b0a9-67a64366ecca" volumeName="kubernetes.io/configmap/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-konnectivity-ca" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199227    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13ea0c7-7fe8-4aec-805c-26197f386653" volumeName="kubernetes.io/projected/a13ea0c7-7fe8-4aec-805c-26197f386653-kube-api-access-7kmqv" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199244    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc4f5277-87b1-43e4-a47e-2a42cc73f400" volumeName="kubernetes.io/empty-dir/dc4f5277-87b1-43e4-a47e-2a42cc73f400-data-volume" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199257    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f463e7-7075-4621-95ae-60c2fefdb950" volumeName="kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199269    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b530358-dea0-471a-85d7-73749d6fa3af" volumeName="kubernetes.io/configmap/3b530358-dea0-471a-85d7-73749d6fa3af-serviceca" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199286    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="639fd403-7cfd-438d-90cb-07c383b61678" volumeName="kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199308    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c203e1-de50-4a9e-a77f-1c7a97335ba1" volumeName="kubernetes.io/configmap/70c203e1-de50-4a9e-a77f-1c7a97335ba1-config-volume" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199323    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c92dc00-2bba-4345-a8c3-b36c72a24056" volumeName="kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199335    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c479a-5489-41fe-b2e4-28f033f0e021" volumeName="kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199347    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199359    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11dab409-45c8-49c5-9300-24db5bfa7a9f" volumeName="kubernetes.io/empty-dir/11dab409-45c8-49c5-9300-24db5bfa7a9f-tmp-dir" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199390    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" volumeName="kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199412    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5088be44-c26e-4b66-b3f0-11b5815d711a" volumeName="kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199423    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="667231b8-a7ce-45ce-adc5-61b1e84c5f12" volumeName="kubernetes.io/projected/667231b8-a7ce-45ce-adc5-61b1e84c5f12-kube-api-access-ct72m" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199439    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76bf1b5b-01e1-4f37-b0a9-67a64366ecca" volumeName="kubernetes.io/secret/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-agent-certs" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199464    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c479a-5489-41fe-b2e4-28f033f0e021" volumeName="kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199477    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6ef7b0f-d31d-41bf-a91e-06c64350ec49" volumeName="kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199488    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13ea0c7-7fe8-4aec-805c-26197f386653" volumeName="kubernetes.io/configmap/a13ea0c7-7fe8-4aec-805c-26197f386653-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199501    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-grpc-tls" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199513    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" volumeName="kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199530    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c402d6b-1936-4cea-a20b-42e4a2ec19a9" volumeName="kubernetes.io/projected/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-kube-api-access-vs7kx" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199547    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199559    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="639fd403-7cfd-438d-90cb-07c383b61678" volumeName="kubernetes.io/projected/639fd403-7cfd-438d-90cb-07c383b61678-kube-api-access-k8fww" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199573    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="667231b8-a7ce-45ce-adc5-61b1e84c5f12" volumeName="kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199585    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c92dc00-2bba-4345-a8c3-b36c72a24056" volumeName="kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-tls" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199598    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e72e536d-221c-43a0-9a31-0062010750ed" volumeName="kubernetes.io/projected/e72e536d-221c-43a0-9a31-0062010750ed-kube-api-access-2qg9l" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199610    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c203e1-de50-4a9e-a77f-1c7a97335ba1" volumeName="kubernetes.io/empty-dir/70c203e1-de50-4a9e-a77f-1c7a97335ba1-tmp-dir" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199625    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199640    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f463e7-7075-4621-95ae-60c2fefdb950" volumeName="kubernetes.io/projected/16f463e7-7075-4621-95ae-60c2fefdb950-kube-api-access-d8xlr" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199654    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="667231b8-a7ce-45ce-adc5-61b1e84c5f12" volumeName="kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-binary-copy" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199719    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b81cb01-44c1-4d62-8f59-443a7f27be7d" volumeName="kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-tls" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199732    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13ea0c7-7fe8-4aec-805c-26197f386653" volumeName="kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199746    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0593c2e4-1a34-445b-8d91-35a1b0888afe" volumeName="kubernetes.io/projected/0593c2e4-1a34-445b-8d91-35a1b0888afe-kube-api-access-gr5hf" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199760    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f463e7-7075-4621-95ae-60c2fefdb950" volumeName="kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-env-overrides" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199773    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" volumeName="kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199785    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf4eb38-7ce4-45fe-8c38-0d8907a8861f" volumeName="kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199796    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c479a-5489-41fe-b2e4-28f033f0e021" volumeName="kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-tls" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199808    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0593c2e4-1a34-445b-8d91-35a1b0888afe" volumeName="kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-cni-binary-copy" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199821    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" volumeName="kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199852    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5088be44-c26e-4b66-b3f0-11b5815d711a" volumeName="kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199868    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c92dc00-2bba-4345-a8c3-b36c72a24056" volumeName="kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199880    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc4f5277-87b1-43e4-a47e-2a42cc73f400" volumeName="kubernetes.io/projected/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-api-access-pcdfb" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199892    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8" volumeName="kubernetes.io/empty-dir/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-tmp" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199903    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5088be44-c26e-4b66-b3f0-11b5815d711a" volumeName="kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199916    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee449a4-59d0-40a5-af37-297952887cc3" volumeName="kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-etc-tuned" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199928    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee449a4-59d0-40a5-af37-297952887cc3" volumeName="kubernetes.io/projected/6ee449a4-59d0-40a5-af37-297952887cc3-kube-api-access-gq5lj" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199942    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c92dc00-2bba-4345-a8c3-b36c72a24056" volumeName="kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199955    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c479a-5489-41fe-b2e4-28f033f0e021" volumeName="kubernetes.io/empty-dir/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-textfile" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199967    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5088be44-c26e-4b66-b3f0-11b5815d711a" volumeName="kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199980    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5088be44-c26e-4b66-b3f0-11b5815d711a" volumeName="kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.199993    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c203e1-de50-4a9e-a77f-1c7a97335ba1" volumeName="kubernetes.io/secret/70c203e1-de50-4a9e-a77f-1c7a97335ba1-metrics-tls" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.200006    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bec7b62-a68f-4579-8bcc-14eb1c26f163" volumeName="kubernetes.io/secret/7bec7b62-a68f-4579-8bcc-14eb1c26f163-cert" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.200018    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13ea0c7-7fe8-4aec-805c-26197f386653" volumeName="kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.200031    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="667231b8-a7ce-45ce-adc5-61b1e84c5f12" volumeName="kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.200043    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c92dc00-2bba-4345-a8c3-b36c72a24056" volumeName="kubernetes.io/projected/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-api-access-cs94g" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.200057    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc4f5277-87b1-43e4-a47e-2a42cc73f400" volumeName="kubernetes.io/configmap/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 01 05:46:03.200025 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.200069    5510 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 05:46:03.202390 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.200076    5510 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 05:46:03.226243 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.226198    5510 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 05:46:03.227918 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.227819    5510 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 05:46:03.227918 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.227921    5510 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 05:46:03.228094 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.227941    5510 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 05:46:03.228094 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.227984    5510 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 05:46:03.242321 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.242283    5510 generic.go:356] "Generic (PLEG): container finished" podID="c39c479a-5489-41fe-b2e4-28f033f0e021" containerID="4fd1f599a4d544bb608190953a17c0bf11fc55d10ad4ce117c73524c430e2d8b" exitCode=0
Nov 01 05:46:03.250944 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.250916    5510 generic.go:356] "Generic (PLEG): container finished" podID="346b48939138ebf5d66bd099d04ae409" containerID="030639c3a91d14d2d6568cb610ee3b1cc6eaede499039fdc88747e372f7e5a70" exitCode=0
Nov 01 05:46:03.280947 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.280919    5510 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="239f24f6dafbfb65b1725de341ba79c0f74629a81c01b72af1202a0193f89a83" exitCode=0
Nov 01 05:46:03.280947 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.280944    5510 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="af970590b9ccf20931af5592390eb67e89cb7fa3d755a6f476d203cb0f9e6718" exitCode=0
Nov 01 05:46:03.280947 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.280951    5510 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="e2e1bdca5160e094ced14d30e9e9f80666d700871e6c31206a7314d0e9ee364e" exitCode=0
Nov 01 05:46:03.281143 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.280957    5510 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="b224925d5d5758cb8aca46c47835c9eace8621b0f6d18edf2935a0dad0895b92" exitCode=0
Nov 01 05:46:03.281143 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.280966    5510 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="894843df3da7b66fd644738cac2b53a228800c69c80c9374d683a7e38d24e74a" exitCode=0
Nov 01 05:46:03.281143 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.280971    5510 generic.go:356] "Generic (PLEG): container finished" podID="667231b8-a7ce-45ce-adc5-61b1e84c5f12" containerID="5f9db22f50c14387c880f80c8ea4807bbe49719d4c66ed0e43883a4ecd6a0f45" exitCode=0
Nov 01 05:46:03.296228 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.296208    5510 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-190.us-west-1.compute.internal\" not found"
Nov 01 05:46:03.304040 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.303994    5510 manager.go:324] Recovery completed
Nov 01 05:46:03.323611 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.323592    5510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:46:03.325511 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.325482    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:46:03.325608 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.325518    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:46:03.325608 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.325529    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:46:03.328802 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.328784    5510 cpu_manager.go:226] "Starting" policy="none"
Nov 01 05:46:03.328802 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.328799    5510 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 05:46:03.328802 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.328802    5510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:46:03.329014 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.328812    5510 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 05:46:03.329014 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.328818    5510 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 05:46:03.329014 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.328983    5510 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 05:46:03.329014 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.328998    5510 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 05:46:03.329196 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329021    5510 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 05:46:03.329196 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329030    5510 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 05:46:03.329196 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329039    5510 policy_none.go:50] "Start"
Nov 01 05:46:03.329196 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329050    5510 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 05:46:03.329196 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329063    5510 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 05:46:03.329426 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329243    5510 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 05:46:03.329426 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329255    5510 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 05:46:03.329426 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329264    5510 policy_none.go:44] "Start"
Nov 01 05:46:03.329903 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329882    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:46:03.329986 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329912    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:46:03.329986 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.329926    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:46:03.333947 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.333920    5510 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 05:46:03.337392 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.337377    5510 manager.go:341] "Starting Device Plugin manager"
Nov 01 05:46:03.337517 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.337401    5510 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 05:46:03.337517 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.337410    5510 server.go:87] "Starting device plugin registration server"
Nov 01 05:46:03.337738 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.337723    5510 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 05:46:03.337801 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.337740    5510 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 05:46:03.337877 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.337803    5510 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 05:46:03.337941 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.337905    5510 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 05:46:03.337991 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.337943    5510 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 05:46:03.337991 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.337950    5510 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 05:46:03.338126 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.338020    5510 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:46:03.337931978Z","UUID":"0a44b20d-25ed-418f-afea-539d1e87b7ef","Handler":null,"Name":"","Endpoint":""}
Nov 01 05:46:03.339512 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.339496    5510 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 05:46:03.340734 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.340713    5510 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 05:46:03.340925 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.339802    5510 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 05:46:03.340992 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.340955    5510 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-190.us-west-1.compute.internal\" not found"
Nov 01 05:46:03.391893 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.391857    5510 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:46:03.392010 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.391894    5510 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:46:03.392226 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.392210    5510 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 05:46:03.438550 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.438525    5510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:46:03.439764 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.439747    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:46:03.439885 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.439775    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:46:03.439885 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.439788    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-190.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:46:03.439885 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.439852    5510 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.490411 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.490385    5510 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:46:03.490531 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.490470    5510 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:46:03.529099 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.529059    5510 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"]
Nov 01 05:46:03.529693 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.529655    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"346b48939138ebf5d66bd099d04ae409","Type":"ContainerStarted","Data":"4657dbccc01a359ad24ee2b07a81ed1cb44543d70a6a9f3926e378bb1f42605d"}
Nov 01 05:46:03.529760 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.529696    5510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.529760 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.529707    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"346b48939138ebf5d66bd099d04ae409","Type":"ContainerDied","Data":"030639c3a91d14d2d6568cb610ee3b1cc6eaede499039fdc88747e372f7e5a70"}
Nov 01 05:46:03.529760 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.529729    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"346b48939138ebf5d66bd099d04ae409","Type":"ContainerStarted","Data":"8fdc637a494aa8ee0816ca41c7c57c4aa15fe98004de388c0bcd81710cabd828"}
Nov 01 05:46:03.529891 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.529784    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"784c45e8dd43338b3f3bbbd833ce9232","Type":"ContainerStarted","Data":"00e1cf3d6069e1b0e0c5c0d01bb926ca13ade8ca5b262d9aa0ec6cb56bef48ec"}
Nov 01 05:46:03.529891 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.529809    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal" event={"ID":"784c45e8dd43338b3f3bbbd833ce9232","Type":"ContainerStarted","Data":"378300804cf7bf6f8296483feb9326b50e13fb9ce944d1d4cf49317be9ffc314"}
Nov 01 05:46:03.529960 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.529921    5510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.550262 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.550242    5510 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.550341 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.550297    5510 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.588683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.588628    5510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:46:03.588683 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.588666    5510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.589193 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.589179    5510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:46:03.589242 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:03.589212    5510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.694188 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.694156    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/784c45e8dd43338b3f3bbbd833ce9232-config\") pod \"kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"784c45e8dd43338b3f3bbbd833ce9232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.694188 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.694189    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.694350 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.694212    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.795148 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.795117    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.795148 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.795146    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/784c45e8dd43338b3f3bbbd833ce9232-config\") pod \"kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"784c45e8dd43338b3f3bbbd833ce9232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.795340 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.795163    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.795340 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.795212    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/784c45e8dd43338b3f3bbbd833ce9232-config\") pod \"kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"784c45e8dd43338b3f3bbbd833ce9232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.795340 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.795219    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:03.795340 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:03.795251    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/346b48939138ebf5d66bd099d04ae409-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" (UID: \"346b48939138ebf5d66bd099d04ae409\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:04.181161 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.181127    5510 apiserver.go:52] "Watching apiserver"
Nov 01 05:46:04.284493 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.284456    5510 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 05:46:04.285073 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.285047    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-r5qzs","openshift-console/console-54986ff55f-nlwd8","openshift-image-registry/node-ca-w7rg6","openshift-ovn-kubernetes/ovnkube-node-gdgnj","openshift-multus/multus-lr295","openshift-cluster-node-tuning-operator/tuned-zcx6z","openshift-network-operator/iptables-alerter-cxzmq","openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78","openshift-multus/multus-additional-cni-plugins-cfp6l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4","openshift-dns/node-resolver-gzvr2","openshift-ingress-canary/ingress-canary-wbwlt","openshift-monitoring/thanos-querier-584b5d845f-c29z8","openshift-console/console-7c87d46cd7-qjbsf","openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m","openshift-multus/network-metrics-daemon-hh9pk","openshift-dns/dns-default-vzzng","openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h","openshift-insights/insights-runtime-extractor-pzwxh","openshift-network-diagnostics/network-check-target-4gj86","openshift-monitoring/node-exporter-w7sbw","kube-system/konnectivity-agent-jmphx","kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"]
Nov 01 05:46:04.295701 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.295672    5510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:04.295821 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.295777    5510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:04.297674 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.297649    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.297763 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.297696    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpkf\" (UniqueName: \"kubernetes.io/projected/70c203e1-de50-4a9e-a77f-1c7a97335ba1-kube-api-access-qmpkf\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:04.297763 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.297752    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-var-lib-kubelet\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.297881 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.297782    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-multus\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.297881 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.297851    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-kubelet\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.297998 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.297891    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fww\" (UniqueName: \"kubernetes.io/projected/639fd403-7cfd-438d-90cb-07c383b61678-kube-api-access-k8fww\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:46:04.297998 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.297920    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-host-slash\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:46:04.297998 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.297973    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-tmp\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.298167 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298006    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-cnibin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.298167 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298055    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-tmp\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.298167 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298066    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-netns\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.298167 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298094    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11dab409-45c8-49c5-9300-24db5bfa7a9f-hosts-file\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:46:04.298167 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298122    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskrb\" (UniqueName: \"kubernetes.io/projected/11dab409-45c8-49c5-9300-24db5bfa7a9f-kube-api-access-tskrb\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:46:04.298167 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298161    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-systemd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.298436 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298286    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-kubelet-config\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:46:04.298436 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298318    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-dbus\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:46:04.298436 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298363    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c203e1-de50-4a9e-a77f-1c7a97335ba1-config-volume\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:04.298436 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298388    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b530358-dea0-471a-85d7-73749d6fa3af-host\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:46:04.298436 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298409    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-conf-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.298436 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298431    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5lj\" (UniqueName: \"kubernetes.io/projected/6ee449a4-59d0-40a5-af37-297952887cc3-kube-api-access-gq5lj\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.298664 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298505    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-multus-certs\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.298664 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298550    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.298664 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298580    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.298664 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298624    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:46:04.298664 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298650    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.298820 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298716    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-log-socket\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.298820 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298733    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qg9l\" (UniqueName: \"kubernetes.io/projected/e72e536d-221c-43a0-9a31-0062010750ed-kube-api-access-2qg9l\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.298917 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298860    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-konnectivity-ca\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:46:04.299002 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.298978    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bec7b62-a68f-4579-8bcc-14eb1c26f163-cert\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:46:04.299125 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299012    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-etc-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.299125 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299034    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc4f5277-87b1-43e4-a47e-2a42cc73f400-crio-socket\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.299125 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299058    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-kubernetes\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.299125 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299083    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-etc-kubernetes\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.299351 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299135    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-system-cni-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.299351 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299158    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-bin\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.299351 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299187    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-netd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.299351 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299219    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.299351 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299255    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-systemd\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.299351 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299284    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc4f5277-87b1-43e4-a47e-2a42cc73f400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.299351 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299310    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299382    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-etc-tuned\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299407    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-node-log\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299431    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16f463e7-7075-4621-95ae-60c2fefdb950-ovn-node-metrics-cert\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299467    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-socket-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299494    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299519    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc4f5277-87b1-43e4-a47e-2a42cc73f400-data-volume\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299547    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7kx\" (UniqueName: \"kubernetes.io/projected/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-kube-api-access-vs7kx\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299572    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrcdv\" (UniqueName: \"kubernetes.io/projected/3b530358-dea0-471a-85d7-73749d6fa3af-kube-api-access-jrcdv\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299608    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-agent-certs\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299634    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-bin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.299683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299675    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cnibin\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299699    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-systemd-units\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299730    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-ovn\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299755    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c203e1-de50-4a9e-a77f-1c7a97335ba1-metrics-tls\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299780    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2sm\" (UniqueName: \"kubernetes.io/projected/6b81cb01-44c1-4d62-8f59-443a7f27be7d-kube-api-access-2m2sm\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299819    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-lib-modules\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.299869    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-script-lib\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300050    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xlr\" (UniqueName: \"kubernetes.io/projected/16f463e7-7075-4621-95ae-60c2fefdb950-kube-api-access-d8xlr\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300105    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc4f5277-87b1-43e4-a47e-2a42cc73f400-data-volume\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.300217 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300129    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ee449a4-59d0-40a5-af37-297952887cc3-etc-tuned\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.300608 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300315    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct72m\" (UniqueName: \"kubernetes.io/projected/667231b8-a7ce-45ce-adc5-61b1e84c5f12-kube-api-access-ct72m\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.300608 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300372    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:46:04.300608 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300457    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:46:04.300608 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300515    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b81cb01-44c1-4d62-8f59-443a7f27be7d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:46:04.300608 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300585    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-run\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.300608 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300607    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5hf\" (UniqueName: \"kubernetes.io/projected/0593c2e4-1a34-445b-8d91-35a1b0888afe-kube-api-access-gr5hf\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300624    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300640    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-slash\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300654    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-config\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300668    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-registration-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300713    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b530358-dea0-471a-85d7-73749d6fa3af-serviceca\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300731    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-modprobe-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300754    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysconfig\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300798    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpz5k\" (UniqueName: \"kubernetes.io/projected/7bec7b62-a68f-4579-8bcc-14eb1c26f163-kube-api-access-lpz5k\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300847    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-socket-dir-parent\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.300880 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300878    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-daemon-config\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300904    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11dab409-45c8-49c5-9300-24db5bfa7a9f-tmp-dir\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300933    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-netns\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300960    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdfb\" (UniqueName: \"kubernetes.io/projected/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-api-access-pcdfb\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300987    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301015    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-conf\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.300991    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11dab409-45c8-49c5-9300-24db5bfa7a9f-tmp-dir\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301041    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-sys\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301077    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-cni-binary-copy\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301096    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-var-lib-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301113    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301132    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-env-overrides\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301202    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c203e1-de50-4a9e-a77f-1c7a97335ba1-tmp-dir\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301232    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-host\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.301285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301266    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-k8s-cni-cncf-io\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301304    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301339    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c203e1-de50-4a9e-a77f-1c7a97335ba1-tmp-dir\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301379    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-device-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301424    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-sys-fs\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301442    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301489    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-iptables-alerter-script\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301524    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-system-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301552    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301578    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-kubelet\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301615    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-hostroot\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.301718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.301634    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.338127 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338101    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.338127 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338111    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzphk\""
Nov 01 05:46:04.338127 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338108    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.338127 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338110    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 05:46:04.338342 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338152    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.338670 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338656    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.338756 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338739    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 05:46:04.338821 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338771    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:46:04.338821 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338799    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tjqf\""
Nov 01 05:46:04.338949 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338850    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v8m4f\""
Nov 01 05:46:04.338996 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338952    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.338996 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.338972    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 05:46:04.339086 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339027    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbc6d\""
Nov 01 05:46:04.339121 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339114    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 05:46:04.339536 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339522    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 05:46:04.339536 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339528    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 05:46:04.339678 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339547    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.339678 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339574    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.339678 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339625    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.339876 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339861    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.339932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339900    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 05:46:04.339932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339911    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 05:46:04.340009 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339929    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 05:46:04.340009 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339944    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2d6d\""
Nov 01 05:46:04.340009 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.339957    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rz4ht\""
Nov 01 05:46:04.340095 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340047    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 05:46:04.340171 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340156    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.340395 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340378    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rpsgx\""
Nov 01 05:46:04.340501 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340486    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:46:04.340567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340523    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-script-lib\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.340567 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340543    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87mzr\""
Nov 01 05:46:04.340628 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340564    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16f463e7-7075-4621-95ae-60c2fefdb950-ovn-node-metrics-cert\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.340756 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340743    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-original-pull-secret\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:46:04.340908 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.340895    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-ovnkube-config\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.341299 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341279    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b530358-dea0-471a-85d7-73749d6fa3af-serviceca\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:46:04.341421 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341406    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16f463e7-7075-4621-95ae-60c2fefdb950-env-overrides\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.341519 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341503    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-cni-binary-copy\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.341550 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341503    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-daemon-config\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.341550 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341540    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639fd403-7cfd-438d-90cb-07c383b61678-metrics-certs\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:46:04.341612 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341546    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.341612 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341509    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 05:46:04.341688 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341676    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-iptables-alerter-script\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:46:04.341763 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341752    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-695z8\""
Nov 01 05:46:04.341846 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341817    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.341876 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.341838    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.349035 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.349021    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.349082 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.349059    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.396855 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.396821    5510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:46:04.396927 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:04.396886    5510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:04.397780 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.397766    5510 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 05:46:04.398132 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.398119    5510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:46:04.398173 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:04.398165    5510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal"
Nov 01 05:46:04.401941 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.401912    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7kx\" (UniqueName: \"kubernetes.io/projected/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-kube-api-access-vs7kx\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:46:04.402058 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.401997    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrcdv\" (UniqueName: \"kubernetes.io/projected/3b530358-dea0-471a-85d7-73749d6fa3af-kube-api-access-jrcdv\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:46:04.402058 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402025    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fww\" (UniqueName: \"kubernetes.io/projected/639fd403-7cfd-438d-90cb-07c383b61678-kube-api-access-k8fww\") pod \"network-metrics-daemon-hh9pk\" (UID: \"639fd403-7cfd-438d-90cb-07c383b61678\") " pod="openshift-multus/network-metrics-daemon-hh9pk"
Nov 01 05:46:04.402193 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402172    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qg9l\" (UniqueName: \"kubernetes.io/projected/e72e536d-221c-43a0-9a31-0062010750ed-kube-api-access-2qg9l\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.402326 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402310    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-multus-certs\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.402387 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402344    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.402387 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402375    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-textfile\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.402493 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402399    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-multus-certs\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.402493 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402409    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c92dc00-2bba-4345-a8c3-b36c72a24056-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.402493 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402428    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.402493 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402438    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402496    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5lj\" (UniqueName: \"kubernetes.io/projected/6ee449a4-59d0-40a5-af37-297952887cc3-kube-api-access-gq5lj\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402516    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c92dc00-2bba-4345-a8c3-b36c72a24056-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402483    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-etc-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402544    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-etc-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402572    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6m62\" (UniqueName: \"kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402537    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-textfile\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402601    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402635    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-system-cni-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.402666 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402665    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402692    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402719    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-systemd\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402728    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-system-cni-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402747    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-node-log\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402749    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402775    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-socket-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402772    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-node-log\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402802    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402794    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-systemd\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402862    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-bin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402873    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402874    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-socket-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402890    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-systemd-units\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402894    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xlr\" (UniqueName: \"kubernetes.io/projected/16f463e7-7075-4621-95ae-60c2fefdb950-kube-api-access-d8xlr\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402913    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-ovn\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402934    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-root\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402940    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-bin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402953    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-metrics-client-ca\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402959    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-systemd-units\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.402972    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-ovn\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403022    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-sys\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.403084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403047    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-metrics-client-ca\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403131    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403165    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403191    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-run\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403209    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403237    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403253    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-var-lib-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403283    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-run\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403287    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-modprobe-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403331    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403343    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysconfig\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403391    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-conf\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403427    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-modprobe-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403446    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-var-lib-openvswitch\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403429    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-sys\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403394    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysconfig\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403474    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-sys\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403485    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-device-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403515    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-sys-fs\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403526    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-device-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403531    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-conf\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403545    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403573    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403580    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-sys-fs\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403603    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-system-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403629    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403655    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403681    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403685    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403683    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5hf\" (UniqueName: \"kubernetes.io/projected/0593c2e4-1a34-445b-8d91-35a1b0888afe-kube-api-access-gr5hf\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403709    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-wtmp\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403681    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-system-cni-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403737    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403755    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-var-lib-kubelet\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403772    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-kubelet\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403794    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-grpc-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403794    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-var-lib-kubelet\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403819    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-host-slash\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403842    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-kubelet\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403882    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c402d6b-1936-4cea-a20b-42e4a2ec19a9-host-slash\") pod \"iptables-alerter-cxzmq\" (UID: \"2c402d6b-1936-4cea-a20b-42e4a2ec19a9\") " pod="openshift-network-operator/iptables-alerter-cxzmq"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403913    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct72m\" (UniqueName: \"kubernetes.io/projected/667231b8-a7ce-45ce-adc5-61b1e84c5f12-kube-api-access-ct72m\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403913    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-netns\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403951    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403980    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.403994    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-netns\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404019    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404046    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404070    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8fs\" (UniqueName: \"kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404096    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b530358-dea0-471a-85d7-73749d6fa3af-host\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404149    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-conf-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404179    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404188    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b530358-dea0-471a-85d7-73749d6fa3af-host\") pod \"node-ca-w7rg6\" (UID: \"3b530358-dea0-471a-85d7-73749d6fa3af\") " pod="openshift-image-registry/node-ca-w7rg6"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404206    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-log-socket\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404227    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-conf-dir\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404234    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404273    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404308    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-log-socket\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404314    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404348    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc4f5277-87b1-43e4-a47e-2a42cc73f400-crio-socket\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404364    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-kubernetes\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404379    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-etc-kubernetes\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404396    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc4f5277-87b1-43e4-a47e-2a42cc73f400-crio-socket\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404393    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-bin\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404400    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-kubernetes\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404428    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-etc-kubernetes\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404430    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-bin\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404441    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-netd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404471    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvph\" (UniqueName: \"kubernetes.io/projected/c39c479a-5489-41fe-b2e4-28f033f0e021-kube-api-access-zdvph\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404496    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404497    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-cni-netd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404527    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404562    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cnibin\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404591    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-lib-modules\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404612    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/667231b8-a7ce-45ce-adc5-61b1e84c5f12-cnibin\") pod \"multus-additional-cni-plugins-cfp6l\" (UID: \"667231b8-a7ce-45ce-adc5-61b1e84c5f12\") " pod="openshift-multus/multus-additional-cni-plugins-cfp6l"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404619    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404649    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404671    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-etc-sysctl-d\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404685    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-lib-modules\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404688    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-slash\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404703    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404713    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-slash\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404714    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-registration-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404749    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72e536d-221c-43a0-9a31-0062010750ed-registration-dir\") pod \"aws-ebs-csi-driver-node-wvtm4\" (UID: \"e72e536d-221c-43a0-9a31-0062010750ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wvtm4"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404756    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404788    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-socket-dir-parent\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404858    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.404924    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-multus-socket-dir-parent\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405275    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-netns\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405292    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-netns\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405319    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-tls\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405346    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13ea0c7-7fe8-4aec-805c-26197f386653-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405372    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2vz\" (UniqueName: \"kubernetes.io/projected/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-kube-api-access-df2vz\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405409    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-host\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405439    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-k8s-cni-cncf-io\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405469    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405488    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ee449a4-59d0-40a5-af37-297952887cc3-host\") pod \"tuned-zcx6z\" (UID: \"6ee449a4-59d0-40a5-af37-297952887cc3\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx6z"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405494    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-kubelet\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405509    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-run-k8s-cni-cncf-io\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405525    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-hostroot\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405539    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-kubelet\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405550    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-hostroot\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405570    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405601    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405609    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-host-run-ovn-kubernetes\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405616    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-tmp\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405646    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-multus\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405666    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-host-var-lib-cni-multus\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405665    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-tmp\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405675    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405708    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmqv\" (UniqueName: \"kubernetes.io/projected/a13ea0c7-7fe8-4aec-805c-26197f386653-kube-api-access-7kmqv\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405737    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-cnibin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405752    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405763    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11dab409-45c8-49c5-9300-24db5bfa7a9f-hosts-file\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405792    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0593c2e4-1a34-445b-8d91-35a1b0888afe-cnibin\") pod \"multus-lr295\" (UID: \"0593c2e4-1a34-445b-8d91-35a1b0888afe\") " pod="openshift-multus/multus-lr295"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405789    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-systemd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405844    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16f463e7-7075-4621-95ae-60c2fefdb950-run-systemd\") pod \"ovnkube-node-gdgnj\" (UID: \"16f463e7-7075-4621-95ae-60c2fefdb950\") " pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405847    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs94g\" (UniqueName: \"kubernetes.io/projected/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-api-access-cs94g\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405871    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11dab409-45c8-49c5-9300-24db5bfa7a9f-hosts-file\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405875    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-kubelet-config\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405916    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-dbus\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405934    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405957    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-dbus\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:46:04.406185 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.405916    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bf4eb38-7ce4-45fe-8c38-0d8907a8861f-kubelet-config\") pod \"global-pull-secret-syncer-r5qzs\" (UID: \"3bf4eb38-7ce4-45fe-8c38-0d8907a8861f\") " pod="kube-system/global-pull-secret-syncer-r5qzs"
Nov 01 05:46:04.439535 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.439474    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.439535 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.439484    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.439535 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.439501    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.439748 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.439601    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 05:46:04.439748 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.439713    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 05:46:04.439748 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.439717    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.439995 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.439954    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 05:46:04.440069 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.439999    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 05:46:04.440130 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440078    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4d6z\""
Nov 01 05:46:04.440182 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440167    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 05:46:04.440234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440217    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74ptq\""
Nov 01 05:46:04.440289 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440235    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 01 05:46:04.440341 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440302    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.440394 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440356    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.440394 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440367    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 05:46:04.440394 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440370    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 05:46:04.440394 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440377    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.440548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440358    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 05:46:04.440548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440416    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.440548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440447    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vp9l2\""
Nov 01 05:46:04.440548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440460    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 05:46:04.440548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440472    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 05:46:04.440548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440449    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 05:46:04.440548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440510    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 05:46:04.440548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440547    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440552    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440567    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440603    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440645    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440678    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440694    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zklw\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440698    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440732    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440768    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7x6xl\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440804    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440797    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 01 05:46:04.440930 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440907    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p56vd\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440896    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440940    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440962    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440966    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440974    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-k6p64\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440955    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dmhjl\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.440997    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f427j\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.441027    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mqtzq\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.441034    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a35uihkrv29ol\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.441057    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.441063    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.441144    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.441229    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc4f5277-87b1-43e4-a47e-2a42cc73f400-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.441251    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 05:46:04.441444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.441286    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qfzvw\""
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442205    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442235    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442262    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442304    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlcv8\""
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442490    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpkf\" (UniqueName: \"kubernetes.io/projected/70c203e1-de50-4a9e-a77f-1c7a97335ba1-kube-api-access-qmpkf\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442501    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdfb\" (UniqueName: \"kubernetes.io/projected/dc4f5277-87b1-43e4-a47e-2a42cc73f400-kube-api-access-pcdfb\") pod \"insights-runtime-extractor-pzwxh\" (UID: \"dc4f5277-87b1-43e4-a47e-2a42cc73f400\") " pod="openshift-insights/insights-runtime-extractor-pzwxh"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442696    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442868    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpz5k\" (UniqueName: \"kubernetes.io/projected/7bec7b62-a68f-4579-8bcc-14eb1c26f163-kube-api-access-lpz5k\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442880    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskrb\" (UniqueName: \"kubernetes.io/projected/11dab409-45c8-49c5-9300-24db5bfa7a9f-kube-api-access-tskrb\") pod \"node-resolver-gzvr2\" (UID: \"11dab409-45c8-49c5-9300-24db5bfa7a9f\") " pod="openshift-dns/node-resolver-gzvr2"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.442902    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443033    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443141    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2sm\" (UniqueName: \"kubernetes.io/projected/6b81cb01-44c1-4d62-8f59-443a7f27be7d-kube-api-access-2m2sm\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443244    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-metrics-client-ca\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443414    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443442    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwm65\" (UniqueName: \"kubernetes.io/projected/e6ef7b0f-d31d-41bf-a91e-06c64350ec49-kube-api-access-bwm65\") pod \"network-check-target-4gj86\" (UID: \"e6ef7b0f-d31d-41bf-a91e-06c64350ec49\") " pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443551    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443580    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-metrics-client-ca\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443742    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.443915    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.444122    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-grpc-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.444326    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-tls\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.444456    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.444607    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.444771    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.444978    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.445131    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.445293    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.445435    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.445587    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13ea0c7-7fe8-4aec-805c-26197f386653-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.445808    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.446109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.445984    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.447545 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.446223    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13ea0c7-7fe8-4aec-805c-26197f386653-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:46:04.447545 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.446284    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.447545 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.446300    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.447545 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.446306    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-tls\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.447545 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.446337    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.447545 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.446349    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.447545 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.446343    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c92dc00-2bba-4345-a8c3-b36c72a24056-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.448594 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.448575    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c203e1-de50-4a9e-a77f-1c7a97335ba1-config-volume\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:04.449725 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.449709    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bec7b62-a68f-4579-8bcc-14eb1c26f163-cert\") pod \"ingress-canary-wbwlt\" (UID: \"7bec7b62-a68f-4579-8bcc-14eb1c26f163\") " pod="openshift-ingress-canary/ingress-canary-wbwlt"
Nov 01 05:46:04.449783 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.449726    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-konnectivity-ca\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:46:04.450392 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.450378    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c203e1-de50-4a9e-a77f-1c7a97335ba1-metrics-tls\") pod \"dns-default-vzzng\" (UID: \"70c203e1-de50-4a9e-a77f-1c7a97335ba1\") " pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:04.450392 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.450384    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76bf1b5b-01e1-4f37-b0a9-67a64366ecca-agent-certs\") pod \"konnectivity-agent-jmphx\" (UID: \"76bf1b5b-01e1-4f37-b0a9-67a64366ecca\") " pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:46:04.450699 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.450683    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b81cb01-44c1-4d62-8f59-443a7f27be7d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:46:04.450758 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.450708    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b81cb01-44c1-4d62-8f59-443a7f27be7d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xvv2h\" (UID: \"6b81cb01-44c1-4d62-8f59-443a7f27be7d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xvv2h"
Nov 01 05:46:04.451256 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.451235    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 05:46:04.455552 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.455535    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.463285 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.463268    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6m62\" (UniqueName: \"kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62\") pod \"console-7c87d46cd7-qjbsf\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") " pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:04.469124 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.469103    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs94g\" (UniqueName: \"kubernetes.io/projected/8c92dc00-2bba-4345-a8c3-b36c72a24056-kube-api-access-cs94g\") pod \"kube-state-metrics-5dc8d7d89b-k8w5m\" (UID: \"8c92dc00-2bba-4345-a8c3-b36c72a24056\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k8w5m"
Nov 01 05:46:04.469229 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.469105    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvph\" (UniqueName: \"kubernetes.io/projected/c39c479a-5489-41fe-b2e4-28f033f0e021-kube-api-access-zdvph\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.469295 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.469262    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmqv\" (UniqueName: \"kubernetes.io/projected/a13ea0c7-7fe8-4aec-805c-26197f386653-kube-api-access-7kmqv\") pod \"openshift-state-metrics-7bd7489957-6kh78\" (UID: \"a13ea0c7-7fe8-4aec-805c-26197f386653\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6kh78"
Nov 01 05:46:04.469480 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.469461    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2vz\" (UniqueName: \"kubernetes.io/projected/09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8-kube-api-access-df2vz\") pod \"thanos-querier-584b5d845f-c29z8\" (UID: \"09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8\") " pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:04.469551 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.469524    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8fs\" (UniqueName: \"kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs\") pod \"console-54986ff55f-nlwd8\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") " pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:04.506421 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.506379    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-root\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.506421 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.506425    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-sys\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.506572 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.506462    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-wtmp\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.506572 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.506481    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-root\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.506572 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.506500    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-sys\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.506572 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.506542    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c39c479a-5489-41fe-b2e4-28f033f0e021-node-exporter-wtmp\") pod \"node-exporter-w7sbw\" (UID: \"c39c479a-5489-41fe-b2e4-28f033f0e021\") " pod="openshift-monitoring/node-exporter-w7sbw"
Nov 01 05:46:04.846486 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.846460    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:04.981923 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:04.981890    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c87d46cd7-qjbsf"]
Nov 01 05:46:05.519310 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:05.519279    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:06.003112 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:06.003087    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vzzng"
Nov 01 05:46:07.302476 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:07.302450    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:07.302923 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:07.302703    5510 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 05:46:09.424056 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:09.424024    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:09.430653 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:09.430628    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-584b5d845f-c29z8"
Nov 01 05:46:09.731865 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:09.730648    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:09.738119 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:09.738087    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:09.914098 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:09.914071    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:46:10.698254 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:10.698214    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:17.543817 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:17.543730    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:46:20.703212 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:20.703189    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:46:28.663479 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:28.663449    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gdgnj"
Nov 01 05:46:30.322610 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.322553    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c87d46cd7-qjbsf" podUID="5088be44-c26e-4b66-b3f0-11b5815d711a" containerName="console" containerID="cri-o://021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a" gracePeriod=15
Nov 01 05:46:30.570525 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.570502    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:30.720844 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.720795    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config\") pod \"5088be44-c26e-4b66-b3f0-11b5815d711a\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") "
Nov 01 05:46:30.721028 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.720863    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca\") pod \"5088be44-c26e-4b66-b3f0-11b5815d711a\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") "
Nov 01 05:46:30.721028 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.720901    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62\" (UniqueName: \"kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62\") pod \"5088be44-c26e-4b66-b3f0-11b5815d711a\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") "
Nov 01 05:46:30.721028 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.720924    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config\") pod \"5088be44-c26e-4b66-b3f0-11b5815d711a\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") "
Nov 01 05:46:30.721028 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.720951    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert\") pod \"5088be44-c26e-4b66-b3f0-11b5815d711a\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") "
Nov 01 05:46:30.721028 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.720984    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp\") pod \"5088be44-c26e-4b66-b3f0-11b5815d711a\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") "
Nov 01 05:46:30.721275 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.721049    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert\") pod \"5088be44-c26e-4b66-b3f0-11b5815d711a\" (UID: \"5088be44-c26e-4b66-b3f0-11b5815d711a\") "
Nov 01 05:46:30.721476 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.721445    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp" pod "5088be44-c26e-4b66-b3f0-11b5815d711a" (UID: "5088be44-c26e-4b66-b3f0-11b5815d711a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:46:30.721597 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.721483    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config" pod "5088be44-c26e-4b66-b3f0-11b5815d711a" (UID: "5088be44-c26e-4b66-b3f0-11b5815d711a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:30.721778 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.721720    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert" pod "5088be44-c26e-4b66-b3f0-11b5815d711a" (UID: "5088be44-c26e-4b66-b3f0-11b5815d711a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:30.721884 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.721862    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca" pod "5088be44-c26e-4b66-b3f0-11b5815d711a" (UID: "5088be44-c26e-4b66-b3f0-11b5815d711a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:30.723054 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.723037    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jmphx"
Nov 01 05:46:30.723517 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.723494    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config" pod "5088be44-c26e-4b66-b3f0-11b5815d711a" (UID: "5088be44-c26e-4b66-b3f0-11b5815d711a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:30.723759 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.723733    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62" pod "5088be44-c26e-4b66-b3f0-11b5815d711a" (UID: "5088be44-c26e-4b66-b3f0-11b5815d711a"). InnerVolumeSpecName "kube-api-access-p6m62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:46:30.723759 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.723739    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert" pod "5088be44-c26e-4b66-b3f0-11b5815d711a" (UID: "5088be44-c26e-4b66-b3f0-11b5815d711a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:30.822244 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.822202    5510 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-oauth-config\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:30.822244 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.822239    5510 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:30.822244 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.822249    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6m62\" (UniqueName: \"kubernetes.io/projected/5088be44-c26e-4b66-b3f0-11b5815d711a-kube-api-access-p6m62\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:30.822244 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.822260    5510 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-console-config\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:30.822491 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.822268    5510 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5088be44-c26e-4b66-b3f0-11b5815d711a-console-serving-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:30.822491 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.822277    5510 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5088be44-c26e-4b66-b3f0-11b5815d711a-tmp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:30.822491 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:30.822286    5510 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5088be44-c26e-4b66-b3f0-11b5815d711a-oauth-serving-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:31.389319 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.389282    5510 generic.go:356] "Generic (PLEG): container finished" podID="5088be44-c26e-4b66-b3f0-11b5815d711a" containerID="021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a" exitCode=0
Nov 01 05:46:31.389745 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.389339    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c87d46cd7-qjbsf" event={"ID":"5088be44-c26e-4b66-b3f0-11b5815d711a","Type":"ContainerDied","Data":"021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a"}
Nov 01 05:46:31.389745 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.389347    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c87d46cd7-qjbsf"
Nov 01 05:46:31.389745 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.389385    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c87d46cd7-qjbsf" event={"ID":"5088be44-c26e-4b66-b3f0-11b5815d711a","Type":"ContainerDied","Data":"673cb71b8aca04aebcab5e38e392dda45a84e0732bc8c1781de7c83b90a61d28"}
Nov 01 05:46:31.389745 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.389404    5510 scope.go:122] "RemoveContainer" containerID="021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a"
Nov 01 05:46:31.397873 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.397845    5510 scope.go:122] "RemoveContainer" containerID="021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a"
Nov 01 05:46:31.398141 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:31.398124    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a\": container with ID starting with 021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a not found: ID does not exist" containerID="021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a"
Nov 01 05:46:31.398201 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.398147    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a"} err="failed to get container status \"021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a\": rpc error: code = NotFound desc = could not find container \"021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a\": container with ID starting with 021f0f693b2996911f0c14203986d15b7d221bd371090fc20901b038defcd36a not found: ID does not exist"
Nov 01 05:46:31.551586 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.551512    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c87d46cd7-qjbsf"]
Nov 01 05:46:31.601157 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:31.601128    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c87d46cd7-qjbsf"]
Nov 01 05:46:33.232984 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:33.232955    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5088be44-c26e-4b66-b3f0-11b5815d711a" path="/var/lib/kubelet/pods/5088be44-c26e-4b66-b3f0-11b5815d711a/volumes"
Nov 01 05:46:39.916504 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:39.916475    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4gj86"
Nov 01 05:46:47.545260 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:46:47.545209    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:46:48.891514 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.891476    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"]
Nov 01 05:46:48.892015 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.891918    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5088be44-c26e-4b66-b3f0-11b5815d711a" containerName="console"
Nov 01 05:46:48.892015 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.891934    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5088be44-c26e-4b66-b3f0-11b5815d711a" containerName="console"
Nov 01 05:46:48.892129 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.892026    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5088be44-c26e-4b66-b3f0-11b5815d711a" containerName="console"
Nov 01 05:46:48.894034 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.894014    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:48.946514 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.946488    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 05:46:48.946514 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.946511    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 05:46:48.946682 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.946548    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 05:46:48.995992 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.995961    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 05:46:48.995992 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.995991    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 05:46:48.996167 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:48.996008    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wjffj\""
Nov 01 05:46:49.046650 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.046624    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 05:46:49.047152 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.047133    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"]
Nov 01 05:46:49.052376 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.052358    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.052429 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.052392    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-metrics-client-ca\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.052429 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.052417    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-secret-telemeter-client\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.052502 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.052472    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6j8\" (UniqueName: \"kubernetes.io/projected/1f330972-954c-4a1b-81e6-223f55770709-kube-api-access-8t6j8\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.052602 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.052587    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.052643 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.052617    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-serving-certs-ca-bundle\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.052681 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.052646    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-telemeter-client-tls\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.052734 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.052722    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-federate-client-tls\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.153863 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.153776    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-secret-telemeter-client\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.153863 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.153806    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6j8\" (UniqueName: \"kubernetes.io/projected/1f330972-954c-4a1b-81e6-223f55770709-kube-api-access-8t6j8\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.154047 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.153936    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.154047 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.153981    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-serving-certs-ca-bundle\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.154047 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.154023    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-telemeter-client-tls\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.154165 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.154066    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-federate-client-tls\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.154165 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.154098    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.154165 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.154141    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-metrics-client-ca\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.154316 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.154193    5510 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 05:46:49.154819 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.154795    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-metrics-client-ca\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.155019 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.154918    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-serving-certs-ca-bundle\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.155140 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.155088    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f330972-954c-4a1b-81e6-223f55770709-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.156720 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.156698    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-federate-client-tls\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.156881 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.156715    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-secret-telemeter-client\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.157049 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.157032    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.157194 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.157175    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1f330972-954c-4a1b-81e6-223f55770709-telemeter-client-tls\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.214199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.214171    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6j8\" (UniqueName: \"kubernetes.io/projected/1f330972-954c-4a1b-81e6-223f55770709-kube-api-access-8t6j8\") pod \"telemeter-client-6d6d64875b-xlxz9\" (UID: \"1f330972-954c-4a1b-81e6-223f55770709\") " pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.506572 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.506545    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"
Nov 01 05:46:49.977093 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.977065    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d6d64875b-xlxz9"]
Nov 01 05:46:49.980734 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:49.980701    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f330972_954c_4a1b_81e6_223f55770709.slice/crio-6da60a00671e3cc7d12271bc850b02aaf4a43917aa9b4088a8eacfd47c0f60ab WatchSource:0}: Error finding container 6da60a00671e3cc7d12271bc850b02aaf4a43917aa9b4088a8eacfd47c0f60ab: Status 404 returned error can't find the container with id 6da60a00671e3cc7d12271bc850b02aaf4a43917aa9b4088a8eacfd47c0f60ab
Nov 01 05:46:49.982538 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:49.982522    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:46:50.453389 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:50.453296    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9" event={"ID":"1f330972-954c-4a1b-81e6-223f55770709","Type":"ContainerStarted","Data":"6da60a00671e3cc7d12271bc850b02aaf4a43917aa9b4088a8eacfd47c0f60ab"}
Nov 01 05:46:52.462084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:52.462046    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9" event={"ID":"1f330972-954c-4a1b-81e6-223f55770709","Type":"ContainerStarted","Data":"22be2d9a023d4efbc714cff2cbddedd945af1a928a8604a1f708202ae752be6a"}
Nov 01 05:46:54.470125 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:54.470087    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9" event={"ID":"1f330972-954c-4a1b-81e6-223f55770709","Type":"ContainerStarted","Data":"0e24e6cc64f40744ecf8081bbfe7d322a304c5260c789dc82fc5161cedacb2c1"}
Nov 01 05:46:54.470539 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:54.470176    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9" event={"ID":"1f330972-954c-4a1b-81e6-223f55770709","Type":"ContainerStarted","Data":"cdaae0673d40f86e9b763bd9a3835ddcdf1b3d1caee98b16a39ba5209b52bb01"}
Nov 01 05:46:54.691080 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:54.691021    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d6d64875b-xlxz9" podStartSLOduration=3.248123166 podStartE2EDuration="6.691003259s" totalImagesPullingTime="3.442880093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:54.641053095 +0000 UTC m=+51.519026506" watchObservedRunningTime="2026-11-01 05:46:54.691003259 +0000 UTC m=+51.568976718"
Nov 01 05:46:55.082698 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.082664    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b444f458-sgqv5"]
Nov 01 05:46:55.085300 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.085278    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.145377 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.145341    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b444f458-sgqv5"]
Nov 01 05:46:55.203866 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.203818    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-service-ca\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.204086 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.204063    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-serving-cert\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.204176 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.204104    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-tmp\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.204176 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.204132    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-trusted-ca-bundle\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.204307 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.204186    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-oauth-serving-cert\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.204307 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.204215    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-oauth-config\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.204307 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.204259    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gr2n\" (UniqueName: \"kubernetes.io/projected/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-kube-api-access-6gr2n\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.204307 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.204289    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-config\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.304721 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.304688    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-oauth-serving-cert\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.304932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.304729    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-oauth-config\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.304932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.304774    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gr2n\" (UniqueName: \"kubernetes.io/projected/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-kube-api-access-6gr2n\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.304932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.304793    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-config\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.304932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.304863    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-service-ca\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.304932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.304884    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-serving-cert\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.304932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.304907    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-tmp\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.304932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.304925    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-trusted-ca-bundle\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.305357 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.305321    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-tmp\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.305504 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.305483    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-oauth-serving-cert\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.305634 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.305606    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-service-ca\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.305971 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.305948    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-config\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.306340 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.306320    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-trusted-ca-bundle\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.307510 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.307488    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-oauth-config\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.307667 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.307650    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-serving-cert\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.365650 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.365591    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gr2n\" (UniqueName: \"kubernetes.io/projected/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-kube-api-access-6gr2n\") pod \"console-84b444f458-sgqv5\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") " pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.397412 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.397389    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:46:55.766366 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:55.766337    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b444f458-sgqv5"]
Nov 01 05:46:55.769453 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:46:55.769427    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeee76cb_6a24_462b_a3ab_d24bfdd3e67d.slice/crio-402d102c81ce55da25c9eb6887c7e1d7d97abd6e35eef30682e094befbf9419d WatchSource:0}: Error finding container 402d102c81ce55da25c9eb6887c7e1d7d97abd6e35eef30682e094befbf9419d: Status 404 returned error can't find the container with id 402d102c81ce55da25c9eb6887c7e1d7d97abd6e35eef30682e094befbf9419d
Nov 01 05:46:56.478590 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:56.478548    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b444f458-sgqv5" event={"ID":"deee76cb-6a24-462b-a3ab-d24bfdd3e67d","Type":"ContainerStarted","Data":"36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02"}
Nov 01 05:46:56.478761 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:56.478598    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b444f458-sgqv5" event={"ID":"deee76cb-6a24-462b-a3ab-d24bfdd3e67d","Type":"ContainerStarted","Data":"402d102c81ce55da25c9eb6887c7e1d7d97abd6e35eef30682e094befbf9419d"}
Nov 01 05:46:56.596530 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:46:56.596478    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84b444f458-sgqv5" podStartSLOduration=1.596464756 podStartE2EDuration="1.596464756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:56.59602754 +0000 UTC m=+53.474000951" watchObservedRunningTime="2026-11-01 05:46:56.596464756 +0000 UTC m=+53.474438146"
Nov 01 05:47:05.398384 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:05.398349    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:47:05.398793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:05.398504    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:47:05.403487 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:05.403464    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:47:05.511052 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:05.511026    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:47:05.670275 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:05.670206    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54986ff55f-nlwd8"]
Nov 01 05:47:15.470360 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.470333    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-795d85c6d9-fr8wc"]
Nov 01 05:47:15.472253 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.472238    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.551282 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.551251    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0cb459-6763-4212-b10f-79e846d8f6b9-tmp\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.551282 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.551282    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-trusted-ca-bundle\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.551457 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.551306    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-serving-cert\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.551457 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.551362    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-config\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.551457 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.551427    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-oauth-config\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.551556 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.551482    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-service-ca\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.551556 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.551506    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-oauth-serving-cert\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.551556 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.551532    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7bl\" (UniqueName: \"kubernetes.io/projected/2a0cb459-6763-4212-b10f-79e846d8f6b9-kube-api-access-cg7bl\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.574369 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.574338    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795d85c6d9-fr8wc"]
Nov 01 05:47:15.652784 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.652747    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-config\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.652784 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.652783    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-oauth-config\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653038 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.652810    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-service-ca\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653038 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.652852    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-oauth-serving-cert\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653038 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.652886    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7bl\" (UniqueName: \"kubernetes.io/projected/2a0cb459-6763-4212-b10f-79e846d8f6b9-kube-api-access-cg7bl\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653038 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.652947    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0cb459-6763-4212-b10f-79e846d8f6b9-tmp\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653038 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.652972    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-trusted-ca-bundle\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653038 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.653029    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-serving-cert\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653603 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.653571    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-config\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653907 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.653884    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-trusted-ca-bundle\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653962 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.653921    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0cb459-6763-4212-b10f-79e846d8f6b9-tmp\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.653962 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.653922    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-oauth-serving-cert\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.654476 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.654455    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0cb459-6763-4212-b10f-79e846d8f6b9-service-ca\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.655470 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.655453    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-oauth-config\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.655740 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.655722    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0cb459-6763-4212-b10f-79e846d8f6b9-console-serving-cert\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.714018 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.713991    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7bl\" (UniqueName: \"kubernetes.io/projected/2a0cb459-6763-4212-b10f-79e846d8f6b9-kube-api-access-cg7bl\") pod \"console-795d85c6d9-fr8wc\" (UID: \"2a0cb459-6763-4212-b10f-79e846d8f6b9\") " pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:15.782180 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:15.782103    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:16.150151 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:16.150079    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795d85c6d9-fr8wc"]
Nov 01 05:47:16.153528 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:47:16.153504    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0cb459_6763_4212_b10f_79e846d8f6b9.slice/crio-d1a493d38bc97a6f3a118c976e7ce00c6b238c912ce2df0f26e44e63033b284b WatchSource:0}: Error finding container d1a493d38bc97a6f3a118c976e7ce00c6b238c912ce2df0f26e44e63033b284b: Status 404 returned error can't find the container with id d1a493d38bc97a6f3a118c976e7ce00c6b238c912ce2df0f26e44e63033b284b
Nov 01 05:47:16.545865 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:16.545810    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795d85c6d9-fr8wc" event={"ID":"2a0cb459-6763-4212-b10f-79e846d8f6b9","Type":"ContainerStarted","Data":"67178b494e950aa2a4329ca49ad1436a1f8cbcdaec322c86f33ad242fb3f2040"}
Nov 01 05:47:16.546251 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:16.545892    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795d85c6d9-fr8wc" event={"ID":"2a0cb459-6763-4212-b10f-79e846d8f6b9","Type":"ContainerStarted","Data":"d1a493d38bc97a6f3a118c976e7ce00c6b238c912ce2df0f26e44e63033b284b"}
Nov 01 05:47:16.663568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:16.663517    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-795d85c6d9-fr8wc" podStartSLOduration=1.663503141 podStartE2EDuration="1.663503141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:47:16.662846456 +0000 UTC m=+73.540819856" watchObservedRunningTime="2026-11-01 05:47:16.663503141 +0000 UTC m=+73.541476548"
Nov 01 05:47:17.542917 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:47:17.542884    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:47:25.782744 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:25.782709    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:25.783226 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:25.782762    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:25.787165 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:25.787146    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:26.580863 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:26.580817    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-795d85c6d9-fr8wc"
Nov 01 05:47:26.748598 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:26.748565    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b444f458-sgqv5"]
Nov 01 05:47:30.690296 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.690259    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-54986ff55f-nlwd8" podUID="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" containerName="console" containerID="cri-o://cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5" gracePeriod=15
Nov 01 05:47:30.938279 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.938257    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:47:30.973427 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.973402    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config\") pod \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") "
Nov 01 05:47:30.973553 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.973431    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle\") pod \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") "
Nov 01 05:47:30.973553 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.973466    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config\") pod \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") "
Nov 01 05:47:30.973663 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.973587    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs\" (UniqueName: \"kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs\") pod \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") "
Nov 01 05:47:30.973663 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.973625    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert\") pod \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") "
Nov 01 05:47:30.973663 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.973653    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp\" (UniqueName: \"kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp\") pod \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") "
Nov 01 05:47:30.973955 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.973926    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle" pod "28ab10be-8c8b-41a0-a6b9-21d43eb30c94" (UID: "28ab10be-8c8b-41a0-a6b9-21d43eb30c94"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:30.974084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.973982    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp" pod "28ab10be-8c8b-41a0-a6b9-21d43eb30c94" (UID: "28ab10be-8c8b-41a0-a6b9-21d43eb30c94"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:47:30.974084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974012    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config" pod "28ab10be-8c8b-41a0-a6b9-21d43eb30c94" (UID: "28ab10be-8c8b-41a0-a6b9-21d43eb30c94"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:30.974084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974038    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca\") pod \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") "
Nov 01 05:47:30.974419 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974108    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert\") pod \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\" (UID: \"28ab10be-8c8b-41a0-a6b9-21d43eb30c94\") "
Nov 01 05:47:30.974419 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974127    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca" pod "28ab10be-8c8b-41a0-a6b9-21d43eb30c94" (UID: "28ab10be-8c8b-41a0-a6b9-21d43eb30c94"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:30.974419 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974303    5510 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-tmp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:30.974419 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974324    5510 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:30.974419 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974342    5510 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-config\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:30.974419 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974354    5510 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-trusted-ca-bundle\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:30.974736 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.974627    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert" pod "28ab10be-8c8b-41a0-a6b9-21d43eb30c94" (UID: "28ab10be-8c8b-41a0-a6b9-21d43eb30c94"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:30.975792 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.975767    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config" pod "28ab10be-8c8b-41a0-a6b9-21d43eb30c94" (UID: "28ab10be-8c8b-41a0-a6b9-21d43eb30c94"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:47:30.975952 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.975933    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert" pod "28ab10be-8c8b-41a0-a6b9-21d43eb30c94" (UID: "28ab10be-8c8b-41a0-a6b9-21d43eb30c94"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:47:30.976006 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:30.975976    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs" pod "28ab10be-8c8b-41a0-a6b9-21d43eb30c94" (UID: "28ab10be-8c8b-41a0-a6b9-21d43eb30c94"). InnerVolumeSpecName "kube-api-access-5z8fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:47:31.075103 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.075072    5510 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-oauth-config\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:31.075103 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.075098    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8fs\" (UniqueName: \"kubernetes.io/projected/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-kube-api-access-5z8fs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:31.075273 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.075112    5510 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-console-serving-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:31.075273 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.075127    5510 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28ab10be-8c8b-41a0-a6b9-21d43eb30c94-oauth-serving-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:31.594851 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.594747    5510 generic.go:356] "Generic (PLEG): container finished" podID="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" containerID="cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5" exitCode=0
Nov 01 05:47:31.594851 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.594794    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54986ff55f-nlwd8"
Nov 01 05:47:31.594851 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.594799    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54986ff55f-nlwd8" event={"ID":"28ab10be-8c8b-41a0-a6b9-21d43eb30c94","Type":"ContainerDied","Data":"cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5"}
Nov 01 05:47:31.595108 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.594868    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54986ff55f-nlwd8" event={"ID":"28ab10be-8c8b-41a0-a6b9-21d43eb30c94","Type":"ContainerDied","Data":"4285f8b10928230dbe192e9c23c19a9e01de42951db6a7cc277c975a6d024711"}
Nov 01 05:47:31.595108 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.594888    5510 scope.go:122] "RemoveContainer" containerID="cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5"
Nov 01 05:47:31.602745 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.602725    5510 scope.go:122] "RemoveContainer" containerID="cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5"
Nov 01 05:47:31.603028 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:47:31.603012    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5\": container with ID starting with cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5 not found: ID does not exist" containerID="cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5"
Nov 01 05:47:31.603080 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.603036    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5"} err="failed to get container status \"cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5\": rpc error: code = NotFound desc = could not find container \"cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5\": container with ID starting with cd61d675c163002ae580e7c41edb56c9cb45b2c0b309ddc367af9553c6bb4ba5 not found: ID does not exist"
Nov 01 05:47:31.757001 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.756965    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54986ff55f-nlwd8"]
Nov 01 05:47:31.806562 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:31.806530    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54986ff55f-nlwd8"]
Nov 01 05:47:33.232558 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:33.232526    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" path="/var/lib/kubelet/pods/28ab10be-8c8b-41a0-a6b9-21d43eb30c94/volumes"
Nov 01 05:47:47.543332 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:47:47.543241    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:47:51.768274 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:51.768215    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84b444f458-sgqv5" podUID="deee76cb-6a24-462b-a3ab-d24bfdd3e67d" containerName="console" containerID="cri-o://36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02" gracePeriod=15
Nov 01 05:47:52.015059 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.015025    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:47:52.130576 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130513    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-serving-cert\") pod \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") "
Nov 01 05:47:52.130576 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130547    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-kube-api-access-6gr2n\" (UniqueName: \"kubernetes.io/projected/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-kube-api-access-6gr2n\") pod \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") "
Nov 01 05:47:52.130772 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130618    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-service-ca\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-service-ca\") pod \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") "
Nov 01 05:47:52.130772 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130637    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-tmp\") pod \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") "
Nov 01 05:47:52.130772 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130671    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-oauth-config\") pod \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") "
Nov 01 05:47:52.130772 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130694    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-config\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-config\") pod \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") "
Nov 01 05:47:52.130772 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130725    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-trusted-ca-bundle\") pod \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") "
Nov 01 05:47:52.130772 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130745    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-oauth-serving-cert\") pod \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\" (UID: \"deee76cb-6a24-462b-a3ab-d24bfdd3e67d\") "
Nov 01 05:47:52.131108 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.130968    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-tmp" pod "deee76cb-6a24-462b-a3ab-d24bfdd3e67d" (UID: "deee76cb-6a24-462b-a3ab-d24bfdd3e67d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:47:52.131170 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.131122    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-service-ca" pod "deee76cb-6a24-462b-a3ab-d24bfdd3e67d" (UID: "deee76cb-6a24-462b-a3ab-d24bfdd3e67d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:52.131385 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.131240    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-oauth-serving-cert" pod "deee76cb-6a24-462b-a3ab-d24bfdd3e67d" (UID: "deee76cb-6a24-462b-a3ab-d24bfdd3e67d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:52.131385 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.131264    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-trusted-ca-bundle" pod "deee76cb-6a24-462b-a3ab-d24bfdd3e67d" (UID: "deee76cb-6a24-462b-a3ab-d24bfdd3e67d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:52.131385 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.131301    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-config" pod "deee76cb-6a24-462b-a3ab-d24bfdd3e67d" (UID: "deee76cb-6a24-462b-a3ab-d24bfdd3e67d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:52.133425 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.133394    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-serving-cert" pod "deee76cb-6a24-462b-a3ab-d24bfdd3e67d" (UID: "deee76cb-6a24-462b-a3ab-d24bfdd3e67d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:47:52.133425 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.133410    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-kube-api-access-6gr2n" pod "deee76cb-6a24-462b-a3ab-d24bfdd3e67d" (UID: "deee76cb-6a24-462b-a3ab-d24bfdd3e67d"). InnerVolumeSpecName "kube-api-access-6gr2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:47:52.133602 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.133575    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-oauth-config" pod "deee76cb-6a24-462b-a3ab-d24bfdd3e67d" (UID: "deee76cb-6a24-462b-a3ab-d24bfdd3e67d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:47:52.231411 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.231382    5510 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:52.231411 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.231407    5510 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-tmp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:52.231571 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.231418    5510 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-oauth-config\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:52.231571 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.231427    5510 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-config\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:52.231571 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.231435    5510 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-trusted-ca-bundle\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:52.231571 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.231443    5510 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-oauth-serving-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:52.231571 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.231451    5510 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-console-serving-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:52.231571 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.231459    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gr2n\" (UniqueName: \"kubernetes.io/projected/deee76cb-6a24-462b-a3ab-d24bfdd3e67d-kube-api-access-6gr2n\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:52.667915 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.667884    5510 generic.go:356] "Generic (PLEG): container finished" podID="deee76cb-6a24-462b-a3ab-d24bfdd3e67d" containerID="36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02" exitCode=0
Nov 01 05:47:52.668076 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.667953    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b444f458-sgqv5"
Nov 01 05:47:52.668076 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.667973    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b444f458-sgqv5" event={"ID":"deee76cb-6a24-462b-a3ab-d24bfdd3e67d","Type":"ContainerDied","Data":"36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02"}
Nov 01 05:47:52.668076 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.668035    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b444f458-sgqv5" event={"ID":"deee76cb-6a24-462b-a3ab-d24bfdd3e67d","Type":"ContainerDied","Data":"402d102c81ce55da25c9eb6887c7e1d7d97abd6e35eef30682e094befbf9419d"}
Nov 01 05:47:52.668076 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.668073    5510 scope.go:122] "RemoveContainer" containerID="36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02"
Nov 01 05:47:52.681541 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.681525    5510 scope.go:122] "RemoveContainer" containerID="36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02"
Nov 01 05:47:52.681801 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:47:52.681783    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02\": container with ID starting with 36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02 not found: ID does not exist" containerID="36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02"
Nov 01 05:47:52.681869 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.681809    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02"} err="failed to get container status \"36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02\": rpc error: code = NotFound desc = could not find container \"36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02\": container with ID starting with 36d5fdb7117a68b86da5020f424b021af05576ee7d4a8093bfcd8b0c4e64ab02 not found: ID does not exist"
Nov 01 05:47:52.830777 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.830750    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b444f458-sgqv5"]
Nov 01 05:47:52.880393 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:52.880364    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b444f458-sgqv5"]
Nov 01 05:47:53.234380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:47:53.234352    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deee76cb-6a24-462b-a3ab-d24bfdd3e67d" path="/var/lib/kubelet/pods/deee76cb-6a24-462b-a3ab-d24bfdd3e67d/volumes"
Nov 01 05:48:17.543382 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:48:17.543335    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:48:47.543448 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:48:47.543408    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:49:17.544330 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:49:17.544251    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:49:47.543343 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:49:47.543305    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:50:17.544168 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:50:17.544129    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:50:26.662631 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.662592    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"]
Nov 01 05:50:26.663247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.662972    5510 cpu_manager.go:414] "containerMap: removing container" podUID="deee76cb-6a24-462b-a3ab-d24bfdd3e67d" containerName="console"
Nov 01 05:50:26.663247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.662991    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deee76cb-6a24-462b-a3ab-d24bfdd3e67d" containerName="console"
Nov 01 05:50:26.663247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.663012    5510 cpu_manager.go:414] "containerMap: removing container" podUID="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" containerName="console"
Nov 01 05:50:26.663247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.663016    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" containerName="console"
Nov 01 05:50:26.663247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.663068    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="deee76cb-6a24-462b-a3ab-d24bfdd3e67d" containerName="console"
Nov 01 05:50:26.663247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.663080    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ab10be-8c8b-41a0-a6b9-21d43eb30c94" containerName="console"
Nov 01 05:50:26.666129 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.666111    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.739453 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.739424    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp52m\" (UniqueName: \"kubernetes.io/projected/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-kube-api-access-jp52m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.739638 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.739464    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.739638 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.739482    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.774302 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.774268    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-h9kfk\""
Nov 01 05:50:26.774302 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.774289    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 05:50:26.774302 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.774276    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 05:50:26.774966 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.774950    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"]
Nov 01 05:50:26.784314 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.784293    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"]
Nov 01 05:50:26.787526 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.787508    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.839355 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.839329    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-69k5d\""
Nov 01 05:50:26.840629 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.840609    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp52m\" (UniqueName: \"kubernetes.io/projected/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-kube-api-access-jp52m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.840689 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.840641    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r76n\" (UniqueName: \"kubernetes.io/projected/e785017f-cdff-45e2-bcc7-0137698a6ef6-kube-api-access-7r76n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.840768 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.840753    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.840897 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.840879    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.840946 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.840931    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.841030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.841013    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.841180 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.841166    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.841246 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.841200    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.854417 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.854386    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"]
Nov 01 05:50:26.901984 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.901948    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp52m\" (UniqueName: \"kubernetes.io/projected/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-kube-api-access-jp52m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:26.941711 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.941623    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.941711 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.941685    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.941942 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.941712    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r76n\" (UniqueName: \"kubernetes.io/projected/e785017f-cdff-45e2-bcc7-0137698a6ef6-kube-api-access-7r76n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.942106 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.942080    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.942149 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.942092    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:26.981662 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:26.981622    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:27.002658 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.002635    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r76n\" (UniqueName: \"kubernetes.io/projected/e785017f-cdff-45e2-bcc7-0137698a6ef6-kube-api-access-7r76n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:27.067956 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.067928    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"]
Nov 01 05:50:27.071255 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.071237    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.105299 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.105271    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:27.124229 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.124206    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 05:50:27.124450 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.124428    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 05:50:27.124539 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.124459    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 05:50:27.124594 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.124553    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 05:50:27.124594 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.124571    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qklrv\""
Nov 01 05:50:27.124594 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.124570    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 05:50:27.124726 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.124629    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 05:50:27.132034 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.132008    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"]
Nov 01 05:50:27.143636 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.143565    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.143757 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.143602    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.143757 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.143673    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.143757 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.143698    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmv8f\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-kube-api-access-vmv8f\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.143757 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.143726    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.143757 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.143757    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0ded5e-e95a-4755-8a26-b3e86c36901b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.143964 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.143776    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.245161 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.245128    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0ded5e-e95a-4755-8a26-b3e86c36901b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.245341 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.245167    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.245341 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.245298    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.245470 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.245354    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.245470 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.245394    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.245470 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.245435    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmv8f\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-kube-api-access-vmv8f\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.245601 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.245481    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.246149 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.246118    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.247602 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.247582    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0ded5e-e95a-4755-8a26-b3e86c36901b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.247722 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.247701    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.247789 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.247741    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.247891 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.247868    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.306370 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.306343    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.306512 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.306348    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmv8f\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-kube-api-access-vmv8f\") pod \"istiod-openshift-gateway-7876bfcb74-tzblc\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.352333 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.352306    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"]
Nov 01 05:50:27.355259 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:50:27.355234    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ac6bc7_e6bc_4d00_8ea9_e0803dbaa205.slice/crio-93e560e0dd73cadba892fd893a370232158ed2f6ec4a987f755eb835bfddcc56 WatchSource:0}: Error finding container 93e560e0dd73cadba892fd893a370232158ed2f6ec4a987f755eb835bfddcc56: Status 404 returned error can't find the container with id 93e560e0dd73cadba892fd893a370232158ed2f6ec4a987f755eb835bfddcc56
Nov 01 05:50:27.382881 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.382858    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:27.475685 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.475655    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"]
Nov 01 05:50:27.478362 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:50:27.478337    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode785017f_cdff_45e2_bcc7_0137698a6ef6.slice/crio-ed68d94c380ecbed3d81378153eec0faae1c5a1fafb674cc8bac6c7836ff9a7a WatchSource:0}: Error finding container ed68d94c380ecbed3d81378153eec0faae1c5a1fafb674cc8bac6c7836ff9a7a: Status 404 returned error can't find the container with id ed68d94c380ecbed3d81378153eec0faae1c5a1fafb674cc8bac6c7836ff9a7a
Nov 01 05:50:27.763768 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:27.763739    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"]
Nov 01 05:50:27.766637 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:50:27.766614    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0ded5e_e95a_4755_8a26_b3e86c36901b.slice/crio-a1061a664c79f94760dbe0759682fb35b759023a0ecdd1e4216b960dd0906686 WatchSource:0}: Error finding container a1061a664c79f94760dbe0759682fb35b759023a0ecdd1e4216b960dd0906686: Status 404 returned error can't find the container with id a1061a664c79f94760dbe0759682fb35b759023a0ecdd1e4216b960dd0906686
Nov 01 05:50:28.187226 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:28.187136    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" event={"ID":"9e0ded5e-e95a-4755-8a26-b3e86c36901b","Type":"ContainerStarted","Data":"a1061a664c79f94760dbe0759682fb35b759023a0ecdd1e4216b960dd0906686"}
Nov 01 05:50:28.188265 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:28.188235    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn" event={"ID":"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205","Type":"ContainerStarted","Data":"93e560e0dd73cadba892fd893a370232158ed2f6ec4a987f755eb835bfddcc56"}
Nov 01 05:50:28.189294 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:28.189271    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f" event={"ID":"e785017f-cdff-45e2-bcc7-0137698a6ef6","Type":"ContainerStarted","Data":"ed68d94c380ecbed3d81378153eec0faae1c5a1fafb674cc8bac6c7836ff9a7a"}
Nov 01 05:50:32.522017 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:32.521978    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:50:32.522385 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:32.522053    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:50:33.215521 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:33.215485    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" event={"ID":"9e0ded5e-e95a-4755-8a26-b3e86c36901b","Type":"ContainerStarted","Data":"899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5"}
Nov 01 05:50:33.215779 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:33.215754    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:33.217456 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:33.217425    5510 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-tzblc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:50:33.217576 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:33.217477    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" podUID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:50:33.338640 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:33.338169    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" podStartSLOduration=1.585569241 podStartE2EDuration="6.338146809s" totalImagesPullingTime="4.752577568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:50:33.337138239 +0000 UTC m=+270.215111661" watchObservedRunningTime="2026-11-01 05:50:33.338146809 +0000 UTC m=+270.216120239"
Nov 01 05:50:34.221791 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:34.221744    5510 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-tzblc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:50:34.222236 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:34.221815    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" podUID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:50:35.227526 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:35.227490    5510 generic.go:356] "Generic (PLEG): container finished" podID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerID="e648e99e9f003dbc82170b8d39f042be6f40042bbc4209379b4ff4417801880c" exitCode=0
Nov 01 05:50:35.228084 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:35.227556    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn" event={"ID":"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205","Type":"ContainerDied","Data":"e648e99e9f003dbc82170b8d39f042be6f40042bbc4209379b4ff4417801880c"}
Nov 01 05:50:36.232629 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:36.232596    5510 generic.go:356] "Generic (PLEG): container finished" podID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerID="6453d9c5ed40813e369de2766acf30ba60cf2653da6794e8743297141797b879" exitCode=0
Nov 01 05:50:36.233042 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:36.232681    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f" event={"ID":"e785017f-cdff-45e2-bcc7-0137698a6ef6","Type":"ContainerDied","Data":"6453d9c5ed40813e369de2766acf30ba60cf2653da6794e8743297141797b879"}
Nov 01 05:50:37.221817 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:37.221781    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:50:38.243364 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:38.243322    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f" event={"ID":"e785017f-cdff-45e2-bcc7-0137698a6ef6","Type":"ContainerDied","Data":"8b22804c11907ef3619c83ec0bef1a8c817f9f6b06273fee1f4b0d47b4157e70"}
Nov 01 05:50:38.243866 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:38.243240    5510 generic.go:356] "Generic (PLEG): container finished" podID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerID="8b22804c11907ef3619c83ec0bef1a8c817f9f6b06273fee1f4b0d47b4157e70" exitCode=0
Nov 01 05:50:39.248734 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:39.248699    5510 generic.go:356] "Generic (PLEG): container finished" podID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerID="43e0db175d5854bd472e26e0595597b7c44e83a89bb4c986110069765cfbe209" exitCode=0
Nov 01 05:50:39.249129 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:39.248770    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn" event={"ID":"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205","Type":"ContainerDied","Data":"43e0db175d5854bd472e26e0595597b7c44e83a89bb4c986110069765cfbe209"}
Nov 01 05:50:46.282633 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:46.282595    5510 generic.go:356] "Generic (PLEG): container finished" podID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerID="4b3cccc7c2bc4ed0e5db3a71b16c8c280059d4aa13c3e9704ce6d30e67c21e3d" exitCode=0
Nov 01 05:50:46.283041 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:46.282677    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f" event={"ID":"e785017f-cdff-45e2-bcc7-0137698a6ef6","Type":"ContainerDied","Data":"4b3cccc7c2bc4ed0e5db3a71b16c8c280059d4aa13c3e9704ce6d30e67c21e3d"}
Nov 01 05:50:46.284788 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:46.284764    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn" event={"ID":"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205","Type":"ContainerStarted","Data":"57f855540c7b7654bc5dbae43781df397f30252f1bf3d16d5d62d6828b5e59d6"}
Nov 01 05:50:46.518265 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:46.518210    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn" podStartSLOduration=2.750855591 podStartE2EDuration="20.518194228s" totalImagesPullingTime="17.767338637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:50:46.517513689 +0000 UTC m=+283.395487124" watchObservedRunningTime="2026-11-01 05:50:46.518194228 +0000 UTC m=+283.396167637"
Nov 01 05:50:47.290879 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.290846    5510 generic.go:356] "Generic (PLEG): container finished" podID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerID="57f855540c7b7654bc5dbae43781df397f30252f1bf3d16d5d62d6828b5e59d6" exitCode=0
Nov 01 05:50:47.291281 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.290925    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn" event={"ID":"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205","Type":"ContainerDied","Data":"57f855540c7b7654bc5dbae43781df397f30252f1bf3d16d5d62d6828b5e59d6"}
Nov 01 05:50:47.411443 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.411420    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:47.538448 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.538415    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-util\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-util\") pod \"e785017f-cdff-45e2-bcc7-0137698a6ef6\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") "
Nov 01 05:50:47.538629 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.538504    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e785017f-cdff-45e2-bcc7-0137698a6ef6-kube-api-access-7r76n\" (UniqueName: \"kubernetes.io/projected/e785017f-cdff-45e2-bcc7-0137698a6ef6-kube-api-access-7r76n\") pod \"e785017f-cdff-45e2-bcc7-0137698a6ef6\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") "
Nov 01 05:50:47.538629 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.538546    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-bundle\") pod \"e785017f-cdff-45e2-bcc7-0137698a6ef6\" (UID: \"e785017f-cdff-45e2-bcc7-0137698a6ef6\") "
Nov 01 05:50:47.539396 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.539365    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-bundle" pod "e785017f-cdff-45e2-bcc7-0137698a6ef6" (UID: "e785017f-cdff-45e2-bcc7-0137698a6ef6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:50:47.540932 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.540872    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e785017f-cdff-45e2-bcc7-0137698a6ef6-kube-api-access-7r76n" pod "e785017f-cdff-45e2-bcc7-0137698a6ef6" (UID: "e785017f-cdff-45e2-bcc7-0137698a6ef6"). InnerVolumeSpecName "kube-api-access-7r76n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:50:47.543270 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.543244    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-util" pod "e785017f-cdff-45e2-bcc7-0137698a6ef6" (UID: "e785017f-cdff-45e2-bcc7-0137698a6ef6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:50:47.546757 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:50:47.546720    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:50:47.639299 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.639277    5510 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-util\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:50:47.639299 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.639300    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r76n\" (UniqueName: \"kubernetes.io/projected/e785017f-cdff-45e2-bcc7-0137698a6ef6-kube-api-access-7r76n\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:50:47.639433 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:47.639310    5510 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e785017f-cdff-45e2-bcc7-0137698a6ef6-bundle\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:50:48.296865 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.296818    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f"
Nov 01 05:50:48.297232 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.296865    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717246f" event={"ID":"e785017f-cdff-45e2-bcc7-0137698a6ef6","Type":"ContainerDied","Data":"ed68d94c380ecbed3d81378153eec0faae1c5a1fafb674cc8bac6c7836ff9a7a"}
Nov 01 05:50:48.297232 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.296912    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed68d94c380ecbed3d81378153eec0faae1c5a1fafb674cc8bac6c7836ff9a7a"
Nov 01 05:50:48.459617 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.459588    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:48.648252 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.648182    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-util\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-util\") pod \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") "
Nov 01 05:50:48.648252 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.648233    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-bundle\") pod \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") "
Nov 01 05:50:48.648463 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.648267    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-kube-api-access-jp52m\" (UniqueName: \"kubernetes.io/projected/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-kube-api-access-jp52m\") pod \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\" (UID: \"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205\") "
Nov 01 05:50:48.648892 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.648869    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-bundle" pod "13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" (UID: "13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:50:48.650515 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.650487    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-kube-api-access-jp52m" pod "13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" (UID: "13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205"). InnerVolumeSpecName "kube-api-access-jp52m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:50:48.652300 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.652283    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-util" pod "13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" (UID: "13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:50:48.749365 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.749334    5510 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-util\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:50:48.749365 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.749361    5510 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-bundle\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:50:48.749514 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:48.749376    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp52m\" (UniqueName: \"kubernetes.io/projected/13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205-kube-api-access-jp52m\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:50:49.302031 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:49.301989    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn" event={"ID":"13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205","Type":"ContainerDied","Data":"93e560e0dd73cadba892fd893a370232158ed2f6ec4a987f755eb835bfddcc56"}
Nov 01 05:50:49.302390 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:49.302041    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfjkwn"
Nov 01 05:50:49.302390 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:50:49.302046    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93e560e0dd73cadba892fd893a370232158ed2f6ec4a987f755eb835bfddcc56"
Nov 01 05:51:07.912586 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912551    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"]
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912838    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerName="extract"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912848    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerName="extract"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912866    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerName="util"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912870    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerName="util"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912878    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerName="pull"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912883    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerName="pull"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912889    5510 cpu_manager.go:414] "containerMap: removing container" podUID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerName="extract"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912893    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerName="extract"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912906    5510 cpu_manager.go:414] "containerMap: removing container" podUID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerName="util"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912911    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerName="util"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912916    5510 cpu_manager.go:414] "containerMap: removing container" podUID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerName="pull"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912920    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerName="pull"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912971    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e785017f-cdff-45e2-bcc7-0137698a6ef6" containerName="extract"
Nov 01 05:51:07.913057 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.912979    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ac6bc7-e6bc-4d00-8ea9-e0803dbaa205" containerName="extract"
Nov 01 05:51:07.915401 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.915386    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:07.987552 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.987523    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/362f43ac-de32-4d3f-9232-9351454d4f5b-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:07.987695 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.987559    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:07.987695 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.987579    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/362f43ac-de32-4d3f-9232-9351454d4f5b-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:07.987695 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.987656    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xvx\" (UniqueName: \"kubernetes.io/projected/362f43ac-de32-4d3f-9232-9351454d4f5b-kube-api-access-24xvx\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:07.987801 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.987739    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:07.987801 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:07.987762    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.042725 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.042686    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 05:51:08.042725 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.042690    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 05:51:08.042725 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.042704    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 05:51:08.042953 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.042762    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 05:51:08.042953 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.042761    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cq826\""
Nov 01 05:51:08.042953 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.042796    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 05:51:08.088835 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.088807    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/362f43ac-de32-4d3f-9232-9351454d4f5b-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.088971 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.088865    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.088971 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.088884    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/362f43ac-de32-4d3f-9232-9351454d4f5b-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.088971 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.088916    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xvx\" (UniqueName: \"kubernetes.io/projected/362f43ac-de32-4d3f-9232-9351454d4f5b-kube-api-access-24xvx\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.088971 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.088936    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.088971 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.088952    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.089344 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.089312    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/362f43ac-de32-4d3f-9232-9351454d4f5b-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.091459 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.091431    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/362f43ac-de32-4d3f-9232-9351454d4f5b-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.091565 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.091512    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.091565 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.091554    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.091684 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.091638    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/362f43ac-de32-4d3f-9232-9351454d4f5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.155530 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.155506    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"]
Nov 01 05:51:08.156904 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.156877    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xvx\" (UniqueName: \"kubernetes.io/projected/362f43ac-de32-4d3f-9232-9351454d4f5b-kube-api-access-24xvx\") pod \"opendatahub-operator-controller-manager-985c476f9-42sfg\" (UID: \"362f43ac-de32-4d3f-9232-9351454d4f5b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.228219 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.228192    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:08.751579 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:08.751543    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"]
Nov 01 05:51:08.755650 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:51:08.755626    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362f43ac_de32_4d3f_9232_9351454d4f5b.slice/crio-14ba056e4d07b4428c6d49396191906a7adc3e349c326bf42d7ef02bbf7130d2 WatchSource:0}: Error finding container 14ba056e4d07b4428c6d49396191906a7adc3e349c326bf42d7ef02bbf7130d2: Status 404 returned error can't find the container with id 14ba056e4d07b4428c6d49396191906a7adc3e349c326bf42d7ef02bbf7130d2
Nov 01 05:51:09.373448 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:09.373412    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg" event={"ID":"362f43ac-de32-4d3f-9232-9351454d4f5b","Type":"ContainerStarted","Data":"14ba056e4d07b4428c6d49396191906a7adc3e349c326bf42d7ef02bbf7130d2"}
Nov 01 05:51:12.387612 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:12.387575    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg" event={"ID":"362f43ac-de32-4d3f-9232-9351454d4f5b","Type":"ContainerStarted","Data":"246f74b45f8f0b652d9d91a5d009487fc6519664da82f504248ddac826d0a404"}
Nov 01 05:51:13.392638 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:13.392603    5510 generic.go:356] "Generic (PLEG): container finished" podID="362f43ac-de32-4d3f-9232-9351454d4f5b" containerID="246f74b45f8f0b652d9d91a5d009487fc6519664da82f504248ddac826d0a404" exitCode=0
Nov 01 05:51:13.393073 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:13.392673    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg" event={"ID":"362f43ac-de32-4d3f-9232-9351454d4f5b","Type":"ContainerDied","Data":"246f74b45f8f0b652d9d91a5d009487fc6519664da82f504248ddac826d0a404"}
Nov 01 05:51:14.398174 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:14.398135    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg" event={"ID":"362f43ac-de32-4d3f-9232-9351454d4f5b","Type":"ContainerStarted","Data":"6891ea4308373bbcb74bda6f728011606464a1688b9b983d40a2bcd826f68864"}
Nov 01 05:51:14.398653 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:14.398263    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:14.632907 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:14.632821    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg" podStartSLOduration=2.972949099 podStartE2EDuration="7.632801929s" totalImagesPullingTime="4.175851232s" totalInitContainerRuntime="484.001598ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:14.583758167 +0000 UTC m=+311.461731577" watchObservedRunningTime="2026-11-01 05:51:14.632801929 +0000 UTC m=+311.510775343"
Nov 01 05:51:17.543951 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:51:17.543918    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:51:25.404030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:25.404004    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-42sfg"
Nov 01 05:51:38.218652 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.218622    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"]
Nov 01 05:51:38.222234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.222217    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"
Nov 01 05:51:38.273953 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.273921    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-b82zw\""
Nov 01 05:51:38.323479 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.323429    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 05:51:38.323479 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.323457    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 05:51:38.325100 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.325074    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"]
Nov 01 05:51:38.340239 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.340215    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dnc\" (UniqueName: \"kubernetes.io/projected/009d88d8-4f7e-49a2-b2eb-a617f8e5cb35-kube-api-access-86dnc\") pod \"cert-manager-cainjector-5c57bbd4fc-qv9np\" (UID: \"009d88d8-4f7e-49a2-b2eb-a617f8e5cb35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"
Nov 01 05:51:38.340351 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.340272    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/009d88d8-4f7e-49a2-b2eb-a617f8e5cb35-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qv9np\" (UID: \"009d88d8-4f7e-49a2-b2eb-a617f8e5cb35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"
Nov 01 05:51:38.440671 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.440644    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dnc\" (UniqueName: \"kubernetes.io/projected/009d88d8-4f7e-49a2-b2eb-a617f8e5cb35-kube-api-access-86dnc\") pod \"cert-manager-cainjector-5c57bbd4fc-qv9np\" (UID: \"009d88d8-4f7e-49a2-b2eb-a617f8e5cb35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"
Nov 01 05:51:38.440805 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.440714    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/009d88d8-4f7e-49a2-b2eb-a617f8e5cb35-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qv9np\" (UID: \"009d88d8-4f7e-49a2-b2eb-a617f8e5cb35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"
Nov 01 05:51:38.503906 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.503848    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/009d88d8-4f7e-49a2-b2eb-a617f8e5cb35-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qv9np\" (UID: \"009d88d8-4f7e-49a2-b2eb-a617f8e5cb35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"
Nov 01 05:51:38.504008 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.503991    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dnc\" (UniqueName: \"kubernetes.io/projected/009d88d8-4f7e-49a2-b2eb-a617f8e5cb35-kube-api-access-86dnc\") pod \"cert-manager-cainjector-5c57bbd4fc-qv9np\" (UID: \"009d88d8-4f7e-49a2-b2eb-a617f8e5cb35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"
Nov 01 05:51:38.546371 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.546345    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"
Nov 01 05:51:38.911603 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:38.911530    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np"]
Nov 01 05:51:38.914847 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:51:38.914801    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod009d88d8_4f7e_49a2_b2eb_a617f8e5cb35.slice/crio-3b6e272e67f65b7e7cbb51ae79dddd3f39534d3a421d53e554e15a9fba52279a WatchSource:0}: Error finding container 3b6e272e67f65b7e7cbb51ae79dddd3f39534d3a421d53e554e15a9fba52279a: Status 404 returned error can't find the container with id 3b6e272e67f65b7e7cbb51ae79dddd3f39534d3a421d53e554e15a9fba52279a
Nov 01 05:51:39.486861 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:39.486810    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np" event={"ID":"009d88d8-4f7e-49a2-b2eb-a617f8e5cb35","Type":"ContainerStarted","Data":"3b6e272e67f65b7e7cbb51ae79dddd3f39534d3a421d53e554e15a9fba52279a"}
Nov 01 05:51:42.501885 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:42.501850    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np" event={"ID":"009d88d8-4f7e-49a2-b2eb-a617f8e5cb35","Type":"ContainerStarted","Data":"c8a6b808aa486b6088f5e49a3075fe37865f6e0d55285aa30dfdf3493f1c908e"}
Nov 01 05:51:42.618013 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:42.617907    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qv9np" podStartSLOduration=1.229492845 podStartE2EDuration="4.617894313s" totalImagesPullingTime="3.388401468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:42.61731685 +0000 UTC m=+339.495290261" watchObservedRunningTime="2026-11-01 05:51:42.617894313 +0000 UTC m=+339.495867723"
Nov 01 05:51:46.070044 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.070009    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"]
Nov 01 05:51:46.073165 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.073145    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.174782 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.174747    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 05:51:46.174968 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.174792    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 05:51:46.174968 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.174853    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 01 05:51:46.174968 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.174890    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 01 05:51:46.174968 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.174899    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9xfz4\""
Nov 01 05:51:46.175696 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.175680    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"]
Nov 01 05:51:46.212737 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.212704    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e89c29e-4c5c-4b3b-91bc-33014e337036-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.212912 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.212784    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e89c29e-4c5c-4b3b-91bc-33014e337036-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.212912 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.212893    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkq4\" (UniqueName: \"kubernetes.io/projected/7e89c29e-4c5c-4b3b-91bc-33014e337036-kube-api-access-nkkq4\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.313822 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.313784    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e89c29e-4c5c-4b3b-91bc-33014e337036-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.314032 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.313864    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkq4\" (UniqueName: \"kubernetes.io/projected/7e89c29e-4c5c-4b3b-91bc-33014e337036-kube-api-access-nkkq4\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.314099 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.314023    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e89c29e-4c5c-4b3b-91bc-33014e337036-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.314627 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.314609    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e89c29e-4c5c-4b3b-91bc-33014e337036-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.316417 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.316392    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e89c29e-4c5c-4b3b-91bc-33014e337036-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.374493 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.374430    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkq4\" (UniqueName: \"kubernetes.io/projected/7e89c29e-4c5c-4b3b-91bc-33014e337036-kube-api-access-nkkq4\") pod \"kuadrant-console-plugin-66c4fb575d-l458f\" (UID: \"7e89c29e-4c5c-4b3b-91bc-33014e337036\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.384277 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.384256    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"
Nov 01 05:51:46.759566 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:46.759539    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f"]
Nov 01 05:51:46.763422 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:51:46.763391    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e89c29e_4c5c_4b3b_91bc_33014e337036.slice/crio-d02ce1eb0c0e1f6b01987a1fbc890d8d6f34a5447fdee427fa16e14fbcebf79d WatchSource:0}: Error finding container d02ce1eb0c0e1f6b01987a1fbc890d8d6f34a5447fdee427fa16e14fbcebf79d: Status 404 returned error can't find the container with id d02ce1eb0c0e1f6b01987a1fbc890d8d6f34a5447fdee427fa16e14fbcebf79d
Nov 01 05:51:47.524973 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:47.524929    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f" event={"ID":"7e89c29e-4c5c-4b3b-91bc-33014e337036","Type":"ContainerStarted","Data":"d02ce1eb0c0e1f6b01987a1fbc890d8d6f34a5447fdee427fa16e14fbcebf79d"}
Nov 01 05:51:47.543448 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:51:47.543406    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:51:52.552444 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.552362    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f" event={"ID":"7e89c29e-4c5c-4b3b-91bc-33014e337036","Type":"ContainerStarted","Data":"34d8f683c5c0ef9e74c9337a5300bcdb93bcd382b3606d5490e65628e2f2fcec"}
Nov 01 05:51:52.666790 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.666733    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l458f" podStartSLOduration=1.222301153 podStartE2EDuration="6.666719704s" totalImagesPullingTime="5.444418551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:52.665922486 +0000 UTC m=+349.543895920" watchObservedRunningTime="2026-11-01 05:51:52.666719704 +0000 UTC m=+349.544693132"
Nov 01 05:51:52.799653 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.799610    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"]
Nov 01 05:51:52.803154 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.803096    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:52.854817 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.854785    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 05:51:52.859801 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.859774    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"]
Nov 01 05:51:52.865079 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.865059    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vv5x\" (UniqueName: \"kubernetes.io/projected/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-kube-api-access-9vv5x\") pod \"limitador-limitador-74dfcfb894-pd6v8\" (UID: \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:52.865194 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.865106    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-config-file\") pod \"limitador-limitador-74dfcfb894-pd6v8\" (UID: \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:52.965475 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.965447    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vv5x\" (UniqueName: \"kubernetes.io/projected/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-kube-api-access-9vv5x\") pod \"limitador-limitador-74dfcfb894-pd6v8\" (UID: \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:52.965641 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.965512    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-config-file\") pod \"limitador-limitador-74dfcfb894-pd6v8\" (UID: \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:52.966215 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:52.966198    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-config-file\") pod \"limitador-limitador-74dfcfb894-pd6v8\" (UID: \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:53.028966 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:53.028936    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vv5x\" (UniqueName: \"kubernetes.io/projected/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-kube-api-access-9vv5x\") pod \"limitador-limitador-74dfcfb894-pd6v8\" (UID: \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:53.051107 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:53.051077    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"]
Nov 01 05:51:53.115663 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:53.115592    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:53.482771 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:53.482744    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"]
Nov 01 05:51:53.485795 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:51:53.485765    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29acc1e_9b3e_49a7_b1fa_ba4d05751cac.slice/crio-d4121ead66f063878345e3fc8a4d4e440125ae618b1a9ac9147ba635d79078f7 WatchSource:0}: Error finding container d4121ead66f063878345e3fc8a4d4e440125ae618b1a9ac9147ba635d79078f7: Status 404 returned error can't find the container with id d4121ead66f063878345e3fc8a4d4e440125ae618b1a9ac9147ba635d79078f7
Nov 01 05:51:53.487849 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:53.487816    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:51:53.557901 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:53.557868    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8" event={"ID":"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac","Type":"ContainerStarted","Data":"d4121ead66f063878345e3fc8a4d4e440125ae618b1a9ac9147ba635d79078f7"}
Nov 01 05:51:54.474453 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:54.474416    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb8bn"]
Nov 01 05:51:54.478747 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:54.478728    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fb8bn"
Nov 01 05:51:54.531694 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:54.531666    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb8bn"]
Nov 01 05:51:54.578359 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:54.578328    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwd2p\" (UniqueName: \"kubernetes.io/projected/36e8817d-e6b4-4ef5-9e8d-c4002d4374f5-kube-api-access-kwd2p\") pod \"authorino-57f7f8659f-fb8bn\" (UID: \"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5\") " pod="kuadrant-system/authorino-57f7f8659f-fb8bn"
Nov 01 05:51:54.580120 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:54.580102    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wgl92\""
Nov 01 05:51:54.679157 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:54.679130    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd2p\" (UniqueName: \"kubernetes.io/projected/36e8817d-e6b4-4ef5-9e8d-c4002d4374f5-kube-api-access-kwd2p\") pod \"authorino-57f7f8659f-fb8bn\" (UID: \"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5\") " pod="kuadrant-system/authorino-57f7f8659f-fb8bn"
Nov 01 05:51:54.742260 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:54.742186    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwd2p\" (UniqueName: \"kubernetes.io/projected/36e8817d-e6b4-4ef5-9e8d-c4002d4374f5-kube-api-access-kwd2p\") pod \"authorino-57f7f8659f-fb8bn\" (UID: \"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5\") " pod="kuadrant-system/authorino-57f7f8659f-fb8bn"
Nov 01 05:51:54.790299 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:54.790264    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fb8bn"
Nov 01 05:51:55.158218 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:55.158191    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb8bn"]
Nov 01 05:51:55.160365 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:51:55.160337    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36e8817d_e6b4_4ef5_9e8d_c4002d4374f5.slice/crio-652b13461b5fa809549d148c486f594f7a21a6da5d79492149c019df6bd722ec WatchSource:0}: Error finding container 652b13461b5fa809549d148c486f594f7a21a6da5d79492149c019df6bd722ec: Status 404 returned error can't find the container with id 652b13461b5fa809549d148c486f594f7a21a6da5d79492149c019df6bd722ec
Nov 01 05:51:55.569310 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:55.569276    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fb8bn" event={"ID":"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5","Type":"ContainerStarted","Data":"652b13461b5fa809549d148c486f594f7a21a6da5d79492149c019df6bd722ec"}
Nov 01 05:51:57.268594 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.268524    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:51:57.272247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.272222    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:57.305170 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.305142    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/def3f09f-380f-4342-9f91-740df732a7fe-config-file\") pod \"limitador-limitador-7c9b87d574-f4wws\" (UID: \"def3f09f-380f-4342-9f91-740df732a7fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:57.305279 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.305176    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd8g5\" (UniqueName: \"kubernetes.io/projected/def3f09f-380f-4342-9f91-740df732a7fe-kube-api-access-bd8g5\") pod \"limitador-limitador-7c9b87d574-f4wws\" (UID: \"def3f09f-380f-4342-9f91-740df732a7fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:57.331076 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.331049    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:51:57.406181 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.406149    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/def3f09f-380f-4342-9f91-740df732a7fe-config-file\") pod \"limitador-limitador-7c9b87d574-f4wws\" (UID: \"def3f09f-380f-4342-9f91-740df732a7fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:57.406326 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.406193    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd8g5\" (UniqueName: \"kubernetes.io/projected/def3f09f-380f-4342-9f91-740df732a7fe-kube-api-access-bd8g5\") pod \"limitador-limitador-7c9b87d574-f4wws\" (UID: \"def3f09f-380f-4342-9f91-740df732a7fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:57.406812 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.406796    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/def3f09f-380f-4342-9f91-740df732a7fe-config-file\") pod \"limitador-limitador-7c9b87d574-f4wws\" (UID: \"def3f09f-380f-4342-9f91-740df732a7fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:57.466414 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.466382    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd8g5\" (UniqueName: \"kubernetes.io/projected/def3f09f-380f-4342-9f91-740df732a7fe-kube-api-access-bd8g5\") pod \"limitador-limitador-7c9b87d574-f4wws\" (UID: \"def3f09f-380f-4342-9f91-740df732a7fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:57.579755 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.579678    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8" event={"ID":"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac","Type":"ContainerStarted","Data":"527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c"}
Nov 01 05:51:57.579921 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.579896    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:51:57.585194 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.585174    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:57.593815 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.593791    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:51:57.695345 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.695298    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8" podStartSLOduration=2.457999813 podStartE2EDuration="5.695283616s" totalImagesPullingTime="3.237283803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:57.694783512 +0000 UTC m=+354.572756924" watchObservedRunningTime="2026-11-01 05:51:57.695283616 +0000 UTC m=+354.573257071"
Nov 01 05:51:57.965006 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:57.964973    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:51:57.978977 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:51:57.978941    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef3f09f_380f_4342_9f91_740df732a7fe.slice/crio-4a72f7df0851759d95b0341fa64b0e43012b59f0d7d18f0b4a7285bc1088973e WatchSource:0}: Error finding container 4a72f7df0851759d95b0341fa64b0e43012b59f0d7d18f0b4a7285bc1088973e: Status 404 returned error can't find the container with id 4a72f7df0851759d95b0341fa64b0e43012b59f0d7d18f0b4a7285bc1088973e
Nov 01 05:51:58.585691 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:58.585652    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fb8bn" event={"ID":"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5","Type":"ContainerStarted","Data":"510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05"}
Nov 01 05:51:58.587233 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:58.587206    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws" event={"ID":"def3f09f-380f-4342-9f91-740df732a7fe","Type":"ContainerStarted","Data":"e648fb2ab6985cd22056b2e94b3a972bd54c66b42485a80e45ecd1800d068487"}
Nov 01 05:51:58.587345 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:58.587261    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws" event={"ID":"def3f09f-380f-4342-9f91-740df732a7fe","Type":"ContainerStarted","Data":"4a72f7df0851759d95b0341fa64b0e43012b59f0d7d18f0b4a7285bc1088973e"}
Nov 01 05:51:58.587345 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:58.587300    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:51:58.824048 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:58.823987    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-fb8bn" podStartSLOduration=1.9312704200000002 podStartE2EDuration="4.823969638s" totalImagesPullingTime="2.892699218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:58.705250985 +0000 UTC m=+355.583224394" watchObservedRunningTime="2026-11-01 05:51:58.823969638 +0000 UTC m=+355.701943050"
Nov 01 05:51:58.824222 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:51:58.824073    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws" podStartSLOduration=1.824069035 podStartE2EDuration="1.824069035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:58.823176054 +0000 UTC m=+355.701149491" watchObservedRunningTime="2026-11-01 05:51:58.824069035 +0000 UTC m=+355.702042444"
Nov 01 05:52:00.719579 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:00.719549    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb8bn"]
Nov 01 05:52:00.720011 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:00.719714    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-fb8bn" podUID="36e8817d-e6b4-4ef5-9e8d-c4002d4374f5" containerName="authorino" containerID="cri-o://510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05" gracePeriod=30
Nov 01 05:52:00.959806 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:00.959778    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fb8bn"
Nov 01 05:52:01.033574 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.033521    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e8817d-e6b4-4ef5-9e8d-c4002d4374f5-kube-api-access-kwd2p\" (UniqueName: \"kubernetes.io/projected/36e8817d-e6b4-4ef5-9e8d-c4002d4374f5-kube-api-access-kwd2p\") pod \"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5\" (UID: \"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5\") "
Nov 01 05:52:01.035714 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.035682    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e8817d-e6b4-4ef5-9e8d-c4002d4374f5-kube-api-access-kwd2p" pod "36e8817d-e6b4-4ef5-9e8d-c4002d4374f5" (UID: "36e8817d-e6b4-4ef5-9e8d-c4002d4374f5"). InnerVolumeSpecName "kube-api-access-kwd2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:52:01.134471 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.134435    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwd2p\" (UniqueName: \"kubernetes.io/projected/36e8817d-e6b4-4ef5-9e8d-c4002d4374f5-kube-api-access-kwd2p\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:52:01.600764 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.600732    5510 generic.go:356] "Generic (PLEG): container finished" podID="36e8817d-e6b4-4ef5-9e8d-c4002d4374f5" containerID="510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05" exitCode=0
Nov 01 05:52:01.600986 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.600781    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fb8bn"
Nov 01 05:52:01.600986 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.600819    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fb8bn" event={"ID":"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5","Type":"ContainerDied","Data":"510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05"}
Nov 01 05:52:01.600986 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.600882    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fb8bn" event={"ID":"36e8817d-e6b4-4ef5-9e8d-c4002d4374f5","Type":"ContainerDied","Data":"652b13461b5fa809549d148c486f594f7a21a6da5d79492149c019df6bd722ec"}
Nov 01 05:52:01.600986 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.600901    5510 scope.go:122] "RemoveContainer" containerID="510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05"
Nov 01 05:52:01.609318 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.609300    5510 scope.go:122] "RemoveContainer" containerID="510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05"
Nov 01 05:52:01.609584 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:52:01.609564    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05\": container with ID starting with 510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05 not found: ID does not exist" containerID="510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05"
Nov 01 05:52:01.609642 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.609596    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05"} err="failed to get container status \"510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05\": rpc error: code = NotFound desc = could not find container \"510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05\": container with ID starting with 510b89ff0cca95c271815d09b687bd6843b4bb995a818f169cb24552e6c60a05 not found: ID does not exist"
Nov 01 05:52:01.762847 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.762806    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb8bn"]
Nov 01 05:52:01.766599 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:01.766577    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb8bn"]
Nov 01 05:52:02.920856 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.920809    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"]
Nov 01 05:52:02.921786 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.921755    5510 cpu_manager.go:414] "containerMap: removing container" podUID="36e8817d-e6b4-4ef5-9e8d-c4002d4374f5" containerName="authorino"
Nov 01 05:52:02.921786 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.921776    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e8817d-e6b4-4ef5-9e8d-c4002d4374f5" containerName="authorino"
Nov 01 05:52:02.921962 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.921866    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e8817d-e6b4-4ef5-9e8d-c4002d4374f5" containerName="authorino"
Nov 01 05:52:02.925963 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.925946    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:02.950404 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.950375    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2201b73-9fec-485b-ba93-ef6a56393bca-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:02.950537 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.950429    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2201b73-9fec-485b-ba93-ef6a56393bca-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:02.950537 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.950448    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2201b73-9fec-485b-ba93-ef6a56393bca-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:02.950633 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.950543    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7r2\" (UniqueName: \"kubernetes.io/projected/d2201b73-9fec-485b-ba93-ef6a56393bca-kube-api-access-dk7r2\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:02.978134 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:02.978103    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lqfrh\""
Nov 01 05:52:03.029093 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.029059    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"]
Nov 01 05:52:03.049239 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.049214    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 05:52:03.051572 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.051553    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2201b73-9fec-485b-ba93-ef6a56393bca-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.051655 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.051606    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2201b73-9fec-485b-ba93-ef6a56393bca-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.051655 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.051624    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2201b73-9fec-485b-ba93-ef6a56393bca-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.051655 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.051646    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7r2\" (UniqueName: \"kubernetes.io/projected/d2201b73-9fec-485b-ba93-ef6a56393bca-kube-api-access-dk7r2\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.051990 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.051974    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2201b73-9fec-485b-ba93-ef6a56393bca-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.052030 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.052021    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2201b73-9fec-485b-ba93-ef6a56393bca-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.077081 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.077058    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 05:52:03.140586 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.140554    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 05:52:03.142496 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.142462    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2201b73-9fec-485b-ba93-ef6a56393bca-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.142962 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.142945    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7r2\" (UniqueName: \"kubernetes.io/projected/d2201b73-9fec-485b-ba93-ef6a56393bca-kube-api-access-dk7r2\") pod \"kserve-module-controller-manager-7566cf5fc8-2vn9w\" (UID: \"d2201b73-9fec-485b-ba93-ef6a56393bca\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.233989 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.233964    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e8817d-e6b4-4ef5-9e8d-c4002d4374f5" path="/var/lib/kubelet/pods/36e8817d-e6b4-4ef5-9e8d-c4002d4374f5/volumes"
Nov 01 05:52:03.288890 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.288865    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lqfrh\""
Nov 01 05:52:03.298469 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.298448    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:03.720060 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:03.720034    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"]
Nov 01 05:52:03.722927 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:52:03.722899    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2201b73_9fec_485b_ba93_ef6a56393bca.slice/crio-7a3051735cb7676692121f1c642029b13354f4034c228dc03a497266a97a5e7a WatchSource:0}: Error finding container 7a3051735cb7676692121f1c642029b13354f4034c228dc03a497266a97a5e7a: Status 404 returned error can't find the container with id 7a3051735cb7676692121f1c642029b13354f4034c228dc03a497266a97a5e7a
Nov 01 05:52:04.613597 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:04.613555    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w" event={"ID":"d2201b73-9fec-485b-ba93-ef6a56393bca","Type":"ContainerStarted","Data":"7a3051735cb7676692121f1c642029b13354f4034c228dc03a497266a97a5e7a"}
Nov 01 05:52:07.629221 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:07.629189    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w" event={"ID":"d2201b73-9fec-485b-ba93-ef6a56393bca","Type":"ContainerStarted","Data":"bbde041c3e3c1c58b980f87e4e6fd5668a36cc01bbea2e5387b82d80c60948aa"}
Nov 01 05:52:07.629631 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:07.629267    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:07.798123 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:07.797436    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w" podStartSLOduration=2.844142156 podStartE2EDuration="5.797419144s" totalImagesPullingTime="2.953276988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:07.797128836 +0000 UTC m=+364.675102246" watchObservedRunningTime="2026-11-01 05:52:07.797419144 +0000 UTC m=+364.675392556"
Nov 01 05:52:08.588039 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:08.588014    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:52:09.593219 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:09.593186    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4wws"
Nov 01 05:52:09.759129 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:09.759094    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"]
Nov 01 05:52:09.759302 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:09.759265    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8" podUID="b29acc1e-9b3e-49a7-b1fa-ba4d05751cac" containerName="limitador" containerID="cri-o://527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c" gracePeriod=30
Nov 01 05:52:10.297484 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.297461    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:52:10.416617 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.416530    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-kube-api-access-9vv5x\" (UniqueName: \"kubernetes.io/projected/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-kube-api-access-9vv5x\") pod \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\" (UID: \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\") "
Nov 01 05:52:10.416617 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.416589    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-config-file\" (UniqueName: \"kubernetes.io/configmap/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-config-file\") pod \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\" (UID: \"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac\") "
Nov 01 05:52:10.417205 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.417108    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-config-file" pod "b29acc1e-9b3e-49a7-b1fa-ba4d05751cac" (UID: "b29acc1e-9b3e-49a7-b1fa-ba4d05751cac"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:52:10.419339 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.419312    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-kube-api-access-9vv5x" pod "b29acc1e-9b3e-49a7-b1fa-ba4d05751cac" (UID: "b29acc1e-9b3e-49a7-b1fa-ba4d05751cac"). InnerVolumeSpecName "kube-api-access-9vv5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:52:10.518016 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.517980    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vv5x\" (UniqueName: \"kubernetes.io/projected/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-kube-api-access-9vv5x\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:52:10.518016 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.518010    5510 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac-config-file\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:52:10.642340 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.642304    5510 generic.go:356] "Generic (PLEG): container finished" podID="b29acc1e-9b3e-49a7-b1fa-ba4d05751cac" containerID="527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c" exitCode=0
Nov 01 05:52:10.642795 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.642363    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"
Nov 01 05:52:10.642795 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.642390    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8" event={"ID":"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac","Type":"ContainerDied","Data":"527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c"}
Nov 01 05:52:10.642795 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.642448    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pd6v8" event={"ID":"b29acc1e-9b3e-49a7-b1fa-ba4d05751cac","Type":"ContainerDied","Data":"d4121ead66f063878345e3fc8a4d4e440125ae618b1a9ac9147ba635d79078f7"}
Nov 01 05:52:10.642795 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.642470    5510 scope.go:122] "RemoveContainer" containerID="527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c"
Nov 01 05:52:10.650735 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.650718    5510 scope.go:122] "RemoveContainer" containerID="527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c"
Nov 01 05:52:10.651038 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:52:10.651020    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c\": container with ID starting with 527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c not found: ID does not exist" containerID="527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c"
Nov 01 05:52:10.651098 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.651047    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c"} err="failed to get container status \"527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c\": rpc error: code = NotFound desc = could not find container \"527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c\": container with ID starting with 527d1a62cbb61117efb8270816f9ff4c8e94bbabc663c370ac232b3bae691e2c not found: ID does not exist"
Nov 01 05:52:10.804966 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.804939    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"]
Nov 01 05:52:10.855125 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:10.855099    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pd6v8"]
Nov 01 05:52:11.233487 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:11.233458    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29acc1e-9b3e-49a7-b1fa-ba4d05751cac" path="/var/lib/kubelet/pods/b29acc1e-9b3e-49a7-b1fa-ba4d05751cac/volumes"
Nov 01 05:52:17.543795 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:52:17.543716    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:52:24.943572 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:24.943530    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"]
Nov 01 05:52:24.944088 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:24.944065    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b29acc1e-9b3e-49a7-b1fa-ba4d05751cac" containerName="limitador"
Nov 01 05:52:24.944131 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:24.944093    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29acc1e-9b3e-49a7-b1fa-ba4d05751cac" containerName="limitador"
Nov 01 05:52:24.944201 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:24.944190    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29acc1e-9b3e-49a7-b1fa-ba4d05751cac" containerName="limitador"
Nov 01 05:52:25.078782 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.078747    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"]
Nov 01 05:52:25.078962 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.078869    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.180664 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.180633    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 05:52:25.180869 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.180787    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 05:52:25.180869 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.180812    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 05:52:25.180869 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.180813    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-xqcp9\""
Nov 01 05:52:25.180869 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.180862    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 05:52:25.181034 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.180863    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:52:25.246925 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.246890    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922d5c31-7006-48a9-95dd-e136446d7a1c-cert\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.246925 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.246923    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/922d5c31-7006-48a9-95dd-e136446d7a1c-manager-config\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.247127 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.246944    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56n5j\" (UniqueName: \"kubernetes.io/projected/922d5c31-7006-48a9-95dd-e136446d7a1c-kube-api-access-56n5j\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.247127 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.246994    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/922d5c31-7006-48a9-95dd-e136446d7a1c-metrics-certs\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.348110 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.348078    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922d5c31-7006-48a9-95dd-e136446d7a1c-cert\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.348110 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.348111    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/922d5c31-7006-48a9-95dd-e136446d7a1c-manager-config\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.348363 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.348133    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n5j\" (UniqueName: \"kubernetes.io/projected/922d5c31-7006-48a9-95dd-e136446d7a1c-kube-api-access-56n5j\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.348363 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.348155    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/922d5c31-7006-48a9-95dd-e136446d7a1c-metrics-certs\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.348816 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.348789    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/922d5c31-7006-48a9-95dd-e136446d7a1c-manager-config\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.350684 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.350664    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922d5c31-7006-48a9-95dd-e136446d7a1c-cert\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.350816 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.350800    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/922d5c31-7006-48a9-95dd-e136446d7a1c-metrics-certs\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.416758 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.416725    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56n5j\" (UniqueName: \"kubernetes.io/projected/922d5c31-7006-48a9-95dd-e136446d7a1c-kube-api-access-56n5j\") pod \"jobset-controller-manager-85c694b95c-tmmsd\" (UID: \"922d5c31-7006-48a9-95dd-e136446d7a1c\") " pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:25.692384 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:25.692311    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:26.064338 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:26.064310    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"]
Nov 01 05:52:26.067359 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:52:26.067328    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922d5c31_7006_48a9_95dd_e136446d7a1c.slice/crio-2a07f309113c349026c1750d2013b46b72e0e4cc152768efd668b4047fc5acc5 WatchSource:0}: Error finding container 2a07f309113c349026c1750d2013b46b72e0e4cc152768efd668b4047fc5acc5: Status 404 returned error can't find the container with id 2a07f309113c349026c1750d2013b46b72e0e4cc152768efd668b4047fc5acc5
Nov 01 05:52:26.708223 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:26.708190    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd" event={"ID":"922d5c31-7006-48a9-95dd-e136446d7a1c","Type":"ContainerStarted","Data":"2a07f309113c349026c1750d2013b46b72e0e4cc152768efd668b4047fc5acc5"}
Nov 01 05:52:28.634556 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:28.634519    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2vn9w"
Nov 01 05:52:30.728267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:30.728232    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd" event={"ID":"922d5c31-7006-48a9-95dd-e136446d7a1c","Type":"ContainerStarted","Data":"46c9aeac9c812d742e72af7ca1d3a9b67e92472f183d32ea47250592c78b4658"}
Nov 01 05:52:30.728662 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:30.728372    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:30.844511 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:30.844461    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd" podStartSLOduration=2.882089445 podStartE2EDuration="6.844446766s" totalImagesPullingTime="3.962357321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:30.843863207 +0000 UTC m=+387.721836617" watchObservedRunningTime="2026-11-01 05:52:30.844446766 +0000 UTC m=+387.722420166"
Nov 01 05:52:41.737464 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:52:41.737434    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-85c694b95c-tmmsd"
Nov 01 05:52:47.544030 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:52:47.543996    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:53:04.263010 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.262974    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"]
Nov 01 05:53:04.270203 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.270172    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.370687 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.370653    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"]
Nov 01 05:53:04.390326 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.390286    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.390783 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.390350    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.390783 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.390446    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.390783 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.390485    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.390783 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.390512    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.390783 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.390755    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqg8z\" (UniqueName: \"kubernetes.io/projected/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-kube-api-access-gqg8z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.391038 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.390786    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.491322 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.491286    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.491322 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.491325    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.491548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.491344    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.491548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.491473    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.491548 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.491518    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.491701 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.491582    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqg8z\" (UniqueName: \"kubernetes.io/projected/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-kube-api-access-gqg8z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.491701 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.491615    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.492614 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.492586    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.494059 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.494033    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.494162 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.494028    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.495103 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.495083    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.495225 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.495206    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.564740 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.564676    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.564919 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.564902    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqg8z\" (UniqueName: \"kubernetes.io/projected/a38250d9-1ebb-43d3-98ff-1b8c19984e8c-kube-api-access-gqg8z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fphbk\" (UID: \"a38250d9-1ebb-43d3-98ff-1b8c19984e8c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:04.585877 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:04.585856    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:05.006691 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:05.006658    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"]
Nov 01 05:53:05.009561 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:53:05.009530    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38250d9_1ebb_43d3_98ff_1b8c19984e8c.slice/crio-ad7188d32e48d62288512e2a347864d8f1babd2eee200e12e83c8da215570c8e WatchSource:0}: Error finding container ad7188d32e48d62288512e2a347864d8f1babd2eee200e12e83c8da215570c8e: Status 404 returned error can't find the container with id ad7188d32e48d62288512e2a347864d8f1babd2eee200e12e83c8da215570c8e
Nov 01 05:53:05.011796 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:05.011761    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:53:05.011909 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:05.011869    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:53:05.883741 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:05.883707    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk" event={"ID":"a38250d9-1ebb-43d3-98ff-1b8c19984e8c","Type":"ContainerStarted","Data":"4568327cf725189580d25d2d4f0f09843cee4925a2cca5da12adada26381390c"}
Nov 01 05:53:05.884161 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:05.883753    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk" event={"ID":"a38250d9-1ebb-43d3-98ff-1b8c19984e8c","Type":"ContainerStarted","Data":"ad7188d32e48d62288512e2a347864d8f1babd2eee200e12e83c8da215570c8e"}
Nov 01 05:53:05.884161 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:05.883857    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:06.053864 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:06.053796    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk" podStartSLOduration=2.053781059 podStartE2EDuration="2.053781059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:06.052667525 +0000 UTC m=+422.930640935" watchObservedRunningTime="2026-11-01 05:53:06.053781059 +0000 UTC m=+422.931754469"
Nov 01 05:53:06.890173 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:06.890138    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fphbk"
Nov 01 05:53:07.112334 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.112298    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"]
Nov 01 05:53:07.112539 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.112519    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" podUID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" containerName="discovery" containerID="cri-o://899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5" gracePeriod=30
Nov 01 05:53:07.221278 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.221252    5510 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-tzblc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:8080/ready\": dial tcp 10.134.0.20:8080: connect: connection refused" start-of-body=
Nov 01 05:53:07.221410 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.221319    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" podUID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.20:8080/ready\": dial tcp 10.134.0.20:8080: connect: connection refused"
Nov 01 05:53:07.360426 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.360402    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:53:07.517632 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.517592    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-cacerts\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-cacerts\") pod \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") "
Nov 01 05:53:07.517819 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.517683    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-dns-cert\") pod \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") "
Nov 01 05:53:07.517819 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.517716    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-token\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-token\") pod \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") "
Nov 01 05:53:07.517819 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.517765    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0ded5e-e95a-4755-8a26-b3e86c36901b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0ded5e-e95a-4755-8a26-b3e86c36901b-local-certs\") pod \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") "
Nov 01 05:53:07.517819 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.517810    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-ca-configmap\") pod \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") "
Nov 01 05:53:07.518059 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.517862    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-kube-api-access-vmv8f\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-kube-api-access-vmv8f\") pod \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") "
Nov 01 05:53:07.518059 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.517914    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-kubeconfig\") pod \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\" (UID: \"9e0ded5e-e95a-4755-8a26-b3e86c36901b\") "
Nov 01 05:53:07.518271 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.518242    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-ca-configmap" pod "9e0ded5e-e95a-4755-8a26-b3e86c36901b" (UID: "9e0ded5e-e95a-4755-8a26-b3e86c36901b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:53:07.520367 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.520337    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-kubeconfig" pod "9e0ded5e-e95a-4755-8a26-b3e86c36901b" (UID: "9e0ded5e-e95a-4755-8a26-b3e86c36901b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:53:07.520547 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.520523    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0ded5e-e95a-4755-8a26-b3e86c36901b-local-certs" pod "9e0ded5e-e95a-4755-8a26-b3e86c36901b" (UID: "9e0ded5e-e95a-4755-8a26-b3e86c36901b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:07.520623 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.520540    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-cacerts" pod "9e0ded5e-e95a-4755-8a26-b3e86c36901b" (UID: "9e0ded5e-e95a-4755-8a26-b3e86c36901b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:53:07.520623 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.520522    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-kube-api-access-vmv8f" pod "9e0ded5e-e95a-4755-8a26-b3e86c36901b" (UID: "9e0ded5e-e95a-4755-8a26-b3e86c36901b"). InnerVolumeSpecName "kube-api-access-vmv8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:07.520702 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.520618    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-token" pod "9e0ded5e-e95a-4755-8a26-b3e86c36901b" (UID: "9e0ded5e-e95a-4755-8a26-b3e86c36901b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:07.520807 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.520787    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-dns-cert" pod "9e0ded5e-e95a-4755-8a26-b3e86c36901b" (UID: "9e0ded5e-e95a-4755-8a26-b3e86c36901b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:53:07.619397 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.619364    5510 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0ded5e-e95a-4755-8a26-b3e86c36901b-local-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:07.619397 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.619391    5510 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-ca-configmap\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:07.619397 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.619400    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmv8f\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-kube-api-access-vmv8f\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:07.619606 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.619409    5510 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-kubeconfig\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:07.619606 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.619418    5510 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-cacerts\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:07.619606 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.619427    5510 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-csr-dns-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:07.619606 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.619435    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0ded5e-e95a-4755-8a26-b3e86c36901b-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:07.893153 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.893073    5510 generic.go:356] "Generic (PLEG): container finished" podID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" containerID="899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5" exitCode=0
Nov 01 05:53:07.893153 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.893119    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"
Nov 01 05:53:07.893643 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.893164    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" event={"ID":"9e0ded5e-e95a-4755-8a26-b3e86c36901b","Type":"ContainerDied","Data":"899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5"}
Nov 01 05:53:07.893643 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.893206    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc" event={"ID":"9e0ded5e-e95a-4755-8a26-b3e86c36901b","Type":"ContainerDied","Data":"a1061a664c79f94760dbe0759682fb35b759023a0ecdd1e4216b960dd0906686"}
Nov 01 05:53:07.893643 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.893231    5510 scope.go:122] "RemoveContainer" containerID="899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5"
Nov 01 05:53:07.902164 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.902149    5510 scope.go:122] "RemoveContainer" containerID="899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5"
Nov 01 05:53:07.902414 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:53:07.902397    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5\": container with ID starting with 899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5 not found: ID does not exist" containerID="899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5"
Nov 01 05:53:07.902476 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:07.902429    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5"} err="failed to get container status \"899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5\": rpc error: code = NotFound desc = could not find container \"899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5\": container with ID starting with 899a38489d6cfa9e6392344d33ee6245568e8f7569f500ae183f37df2eda60e5 not found: ID does not exist"
Nov 01 05:53:08.056232 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:08.056201    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"]
Nov 01 05:53:08.105527 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:08.105500    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tzblc"]
Nov 01 05:53:09.234233 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:09.234201    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" path="/var/lib/kubelet/pods/9e0ded5e-e95a-4755-8a26-b3e86c36901b/volumes"
Nov 01 05:53:17.547301 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:53:17.547263    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:53:26.927982 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:26.927944    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:53:36.460157 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.460121    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-696f7c4d5c-6px8t"]
Nov 01 05:53:36.460576 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.460459    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" containerName="discovery"
Nov 01 05:53:36.460576 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.460471    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" containerName="discovery"
Nov 01 05:53:36.460576 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.460538    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0ded5e-e95a-4755-8a26-b3e86c36901b" containerName="discovery"
Nov 01 05:53:36.463456 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.463439    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:53:36.560510 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.560482    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2465j\" (UniqueName: \"kubernetes.io/projected/d211e9b9-226b-422c-9408-7a1690a1064f-kube-api-access-2465j\") pod \"authorino-696f7c4d5c-6px8t\" (UID: \"d211e9b9-226b-422c-9408-7a1690a1064f\") " pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:53:36.560669 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.560536    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d211e9b9-226b-422c-9408-7a1690a1064f-tls-cert\") pod \"authorino-696f7c4d5c-6px8t\" (UID: \"d211e9b9-226b-422c-9408-7a1690a1064f\") " pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:53:36.565254 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.565232    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 05:53:36.565380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.565340    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wgl92\""
Nov 01 05:53:36.566389 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.566374    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-696f7c4d5c-6px8t"]
Nov 01 05:53:36.661574 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.661537    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d211e9b9-226b-422c-9408-7a1690a1064f-tls-cert\") pod \"authorino-696f7c4d5c-6px8t\" (UID: \"d211e9b9-226b-422c-9408-7a1690a1064f\") " pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:53:36.661756 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.661612    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2465j\" (UniqueName: \"kubernetes.io/projected/d211e9b9-226b-422c-9408-7a1690a1064f-kube-api-access-2465j\") pod \"authorino-696f7c4d5c-6px8t\" (UID: \"d211e9b9-226b-422c-9408-7a1690a1064f\") " pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:53:36.668447 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.668417    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d211e9b9-226b-422c-9408-7a1690a1064f-tls-cert\") pod \"authorino-696f7c4d5c-6px8t\" (UID: \"d211e9b9-226b-422c-9408-7a1690a1064f\") " pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:53:36.723306 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.723238    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2465j\" (UniqueName: \"kubernetes.io/projected/d211e9b9-226b-422c-9408-7a1690a1064f-kube-api-access-2465j\") pod \"authorino-696f7c4d5c-6px8t\" (UID: \"d211e9b9-226b-422c-9408-7a1690a1064f\") " pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:53:36.775374 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:36.775332    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:53:37.147032 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:37.147004    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-696f7c4d5c-6px8t"]
Nov 01 05:53:37.150004 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:53:37.149976    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd211e9b9_226b_422c_9408_7a1690a1064f.slice/crio-ab52fdf779c6cc7ee9f56ff14af3f6bb1ea74c77b0615f3121c7e2fd0f16d262 WatchSource:0}: Error finding container ab52fdf779c6cc7ee9f56ff14af3f6bb1ea74c77b0615f3121c7e2fd0f16d262: Status 404 returned error can't find the container with id ab52fdf779c6cc7ee9f56ff14af3f6bb1ea74c77b0615f3121c7e2fd0f16d262
Nov 01 05:53:38.021848 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:38.021793    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-696f7c4d5c-6px8t" event={"ID":"d211e9b9-226b-422c-9408-7a1690a1064f","Type":"ContainerStarted","Data":"ab52fdf779c6cc7ee9f56ff14af3f6bb1ea74c77b0615f3121c7e2fd0f16d262"}
Nov 01 05:53:39.027256 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:39.027223    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-696f7c4d5c-6px8t" event={"ID":"d211e9b9-226b-422c-9408-7a1690a1064f","Type":"ContainerStarted","Data":"9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7"}
Nov 01 05:53:39.142347 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:53:39.142287    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-696f7c4d5c-6px8t" podStartSLOduration=1.852755878 podStartE2EDuration="3.142266768s" totalImagesPullingTime="1.28951089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:39.141561682 +0000 UTC m=+456.019535093" watchObservedRunningTime="2026-11-01 05:53:39.142266768 +0000 UTC m=+456.020240178"
Nov 01 05:53:47.543675 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:53:47.543592    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:54:17.543923 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:54:17.543884    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:54:47.544543 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:54:47.544501    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:54:54.674284 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.674250    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6jlp8"]
Nov 01 05:54:54.677487 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.677471    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:54.782209 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.782177    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 01 05:54:54.782389 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.782256    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-77rtc\""
Nov 01 05:54:54.782389 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.782283    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 01 05:54:54.783338 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.783318    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6jlp8"]
Nov 01 05:54:54.816680 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.816651    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd04cd64-3279-48c2-98e7-84b71aca1188-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6jlp8\" (UID: \"fd04cd64-3279-48c2-98e7-84b71aca1188\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:54.816809 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.816720    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2sp\" (UniqueName: \"kubernetes.io/projected/fd04cd64-3279-48c2-98e7-84b71aca1188-kube-api-access-ck2sp\") pod \"ai-gateway-controller-7d85675f98-6jlp8\" (UID: \"fd04cd64-3279-48c2-98e7-84b71aca1188\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:54.917165 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.917121    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd04cd64-3279-48c2-98e7-84b71aca1188-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6jlp8\" (UID: \"fd04cd64-3279-48c2-98e7-84b71aca1188\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:54.917339 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.917190    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2sp\" (UniqueName: \"kubernetes.io/projected/fd04cd64-3279-48c2-98e7-84b71aca1188-kube-api-access-ck2sp\") pod \"ai-gateway-controller-7d85675f98-6jlp8\" (UID: \"fd04cd64-3279-48c2-98e7-84b71aca1188\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:54.919619 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.919599    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd04cd64-3279-48c2-98e7-84b71aca1188-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6jlp8\" (UID: \"fd04cd64-3279-48c2-98e7-84b71aca1188\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:54.977794 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.977771    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2sp\" (UniqueName: \"kubernetes.io/projected/fd04cd64-3279-48c2-98e7-84b71aca1188-kube-api-access-ck2sp\") pod \"ai-gateway-controller-7d85675f98-6jlp8\" (UID: \"fd04cd64-3279-48c2-98e7-84b71aca1188\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:54.990646 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:54.990623    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:55.359582 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:55.359488    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6jlp8"]
Nov 01 05:54:55.362229 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:54:55.362193    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd04cd64_3279_48c2_98e7_84b71aca1188.slice/crio-2686b9789414652dc3a462479768f0a1f5c5e87b679089736e2315a41f679adb WatchSource:0}: Error finding container 2686b9789414652dc3a462479768f0a1f5c5e87b679089736e2315a41f679adb: Status 404 returned error can't find the container with id 2686b9789414652dc3a462479768f0a1f5c5e87b679089736e2315a41f679adb
Nov 01 05:54:56.363638 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:56.363599    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8" event={"ID":"fd04cd64-3279-48c2-98e7-84b71aca1188","Type":"ContainerStarted","Data":"2686b9789414652dc3a462479768f0a1f5c5e87b679089736e2315a41f679adb"}
Nov 01 05:54:59.378678 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:59.378641    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8" event={"ID":"fd04cd64-3279-48c2-98e7-84b71aca1188","Type":"ContainerStarted","Data":"0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8"}
Nov 01 05:54:59.379078 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:59.378743    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:54:59.494425 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:54:59.494370    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8" podStartSLOduration=2.384306978 podStartE2EDuration="5.494355989s" totalImagesPullingTime="3.110049011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:54:59.493372861 +0000 UTC m=+536.371346276" watchObservedRunningTime="2026-11-01 05:54:59.494355989 +0000 UTC m=+536.372329399"
Nov 01 05:55:01.563954 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.563918    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"]
Nov 01 05:55:01.567335 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.567317    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:01.619301 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.619259    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 05:55:01.668973 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.668939    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7d8fb\""
Nov 01 05:55:01.670015 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.669991    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"]
Nov 01 05:55:01.691606 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.691575    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a714429e-0dca-453f-bb85-d62bef7ad6ad-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dzmqd\" (UID: \"a714429e-0dca-453f-bb85-d62bef7ad6ad\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:01.691778 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.691624    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9r6\" (UniqueName: \"kubernetes.io/projected/a714429e-0dca-453f-bb85-d62bef7ad6ad-kube-api-access-8v9r6\") pod \"payload-pre-processing-7b98bb678c-dzmqd\" (UID: \"a714429e-0dca-453f-bb85-d62bef7ad6ad\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:01.792661 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.792617    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a714429e-0dca-453f-bb85-d62bef7ad6ad-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dzmqd\" (UID: \"a714429e-0dca-453f-bb85-d62bef7ad6ad\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:01.792881 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.792684    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9r6\" (UniqueName: \"kubernetes.io/projected/a714429e-0dca-453f-bb85-d62bef7ad6ad-kube-api-access-8v9r6\") pod \"payload-pre-processing-7b98bb678c-dzmqd\" (UID: \"a714429e-0dca-453f-bb85-d62bef7ad6ad\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:01.793473 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.793455    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a714429e-0dca-453f-bb85-d62bef7ad6ad-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dzmqd\" (UID: \"a714429e-0dca-453f-bb85-d62bef7ad6ad\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:01.853755 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.853672    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9r6\" (UniqueName: \"kubernetes.io/projected/a714429e-0dca-453f-bb85-d62bef7ad6ad-kube-api-access-8v9r6\") pod \"payload-pre-processing-7b98bb678c-dzmqd\" (UID: \"a714429e-0dca-453f-bb85-d62bef7ad6ad\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:01.880527 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:01.880494    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:02.248984 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:02.248957    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"]
Nov 01 05:55:02.251536 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:55:02.251510    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda714429e_0dca_453f_bb85_d62bef7ad6ad.slice/crio-d50bacb0ff7ac1e7c66840f5e9a6bacfe3251ba59863d3d74080b8b1ce8f741b WatchSource:0}: Error finding container d50bacb0ff7ac1e7c66840f5e9a6bacfe3251ba59863d3d74080b8b1ce8f741b: Status 404 returned error can't find the container with id d50bacb0ff7ac1e7c66840f5e9a6bacfe3251ba59863d3d74080b8b1ce8f741b
Nov 01 05:55:02.390962 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:02.390929    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd" event={"ID":"a714429e-0dca-453f-bb85-d62bef7ad6ad","Type":"ContainerStarted","Data":"d50bacb0ff7ac1e7c66840f5e9a6bacfe3251ba59863d3d74080b8b1ce8f741b"}
Nov 01 05:55:05.405710 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:05.405610    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd" event={"ID":"a714429e-0dca-453f-bb85-d62bef7ad6ad","Type":"ContainerStarted","Data":"4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78"}
Nov 01 05:55:05.406198 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:05.405745    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:05.544718 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:05.544661    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd" podStartSLOduration=1.737600986 podStartE2EDuration="4.544647682s" totalImagesPullingTime="2.807046696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:05.52209066 +0000 UTC m=+542.400064071" watchObservedRunningTime="2026-11-01 05:55:05.544647682 +0000 UTC m=+542.422621092"
Nov 01 05:55:10.384442 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.384411    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:55:10.735518 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.735484    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7684cd596b-68gdk"]
Nov 01 05:55:10.739032 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.739012    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:10.774192 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.774158    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05da642c-8901-434c-859a-ad7386a0302c-metrics-certs\") pod \"ai-gateway-controller-7684cd596b-68gdk\" (UID: \"05da642c-8901-434c-859a-ad7386a0302c\") " pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:10.774368 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.774202    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4v8q\" (UniqueName: \"kubernetes.io/projected/05da642c-8901-434c-859a-ad7386a0302c-kube-api-access-w4v8q\") pod \"ai-gateway-controller-7684cd596b-68gdk\" (UID: \"05da642c-8901-434c-859a-ad7386a0302c\") " pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:10.796549 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.796514    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7684cd596b-68gdk"]
Nov 01 05:55:10.875153 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.875106    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05da642c-8901-434c-859a-ad7386a0302c-metrics-certs\") pod \"ai-gateway-controller-7684cd596b-68gdk\" (UID: \"05da642c-8901-434c-859a-ad7386a0302c\") " pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:10.875153 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.875155    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4v8q\" (UniqueName: \"kubernetes.io/projected/05da642c-8901-434c-859a-ad7386a0302c-kube-api-access-w4v8q\") pod \"ai-gateway-controller-7684cd596b-68gdk\" (UID: \"05da642c-8901-434c-859a-ad7386a0302c\") " pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:10.877934 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.877908    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05da642c-8901-434c-859a-ad7386a0302c-metrics-certs\") pod \"ai-gateway-controller-7684cd596b-68gdk\" (UID: \"05da642c-8901-434c-859a-ad7386a0302c\") " pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:10.936248 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:10.936212    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4v8q\" (UniqueName: \"kubernetes.io/projected/05da642c-8901-434c-859a-ad7386a0302c-kube-api-access-w4v8q\") pod \"ai-gateway-controller-7684cd596b-68gdk\" (UID: \"05da642c-8901-434c-859a-ad7386a0302c\") " pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:11.051615 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:11.051531    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:11.422107 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:11.422024    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7684cd596b-68gdk"]
Nov 01 05:55:11.425453 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:55:11.425421    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05da642c_8901_434c_859a_ad7386a0302c.slice/crio-ff680765367066c296dcd10e166ada2e913d5e703bf644e747b7e2451c8ca27e WatchSource:0}: Error finding container ff680765367066c296dcd10e166ada2e913d5e703bf644e747b7e2451c8ca27e: Status 404 returned error can't find the container with id ff680765367066c296dcd10e166ada2e913d5e703bf644e747b7e2451c8ca27e
Nov 01 05:55:11.431617 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:11.431584    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk" event={"ID":"05da642c-8901-434c-859a-ad7386a0302c","Type":"ContainerStarted","Data":"ff680765367066c296dcd10e166ada2e913d5e703bf644e747b7e2451c8ca27e"}
Nov 01 05:55:12.437678 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:12.437644    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk" event={"ID":"05da642c-8901-434c-859a-ad7386a0302c","Type":"ContainerStarted","Data":"494f84bedb15cc2a902521aa940b428a284942ed16a48ccf2fc56e12dc84f62e"}
Nov 01 05:55:12.438182 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:12.437770    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:12.554114 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:12.554002    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk" podStartSLOduration=1.7449777050000002 podStartE2EDuration="2.55398756s" totalImagesPullingTime="809.009855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:12.552756976 +0000 UTC m=+549.430730388" watchObservedRunningTime="2026-11-01 05:55:12.55398756 +0000 UTC m=+549.431960969"
Nov 01 05:55:16.413193 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:16.413118    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 05:55:17.544057 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:55:17.544020    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:55:23.445304 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.445274    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7684cd596b-68gdk"
Nov 01 05:55:23.614037 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.613996    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6jlp8"]
Nov 01 05:55:23.614328 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.614296    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8" podUID="fd04cd64-3279-48c2-98e7-84b71aca1188" containerName="manager" containerID="cri-o://0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8" gracePeriod=30
Nov 01 05:55:23.853686 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.853663    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:55:23.890458 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.890426    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd04cd64-3279-48c2-98e7-84b71aca1188-kube-api-access-ck2sp\" (UniqueName: \"kubernetes.io/projected/fd04cd64-3279-48c2-98e7-84b71aca1188-kube-api-access-ck2sp\") pod \"fd04cd64-3279-48c2-98e7-84b71aca1188\" (UID: \"fd04cd64-3279-48c2-98e7-84b71aca1188\") "
Nov 01 05:55:23.890591 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.890490    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd04cd64-3279-48c2-98e7-84b71aca1188-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd04cd64-3279-48c2-98e7-84b71aca1188-metrics-certs\") pod \"fd04cd64-3279-48c2-98e7-84b71aca1188\" (UID: \"fd04cd64-3279-48c2-98e7-84b71aca1188\") "
Nov 01 05:55:23.892761 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.892726    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd04cd64-3279-48c2-98e7-84b71aca1188-kube-api-access-ck2sp" pod "fd04cd64-3279-48c2-98e7-84b71aca1188" (UID: "fd04cd64-3279-48c2-98e7-84b71aca1188"). InnerVolumeSpecName "kube-api-access-ck2sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:55:23.892910 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.892894    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd04cd64-3279-48c2-98e7-84b71aca1188-metrics-certs" pod "fd04cd64-3279-48c2-98e7-84b71aca1188" (UID: "fd04cd64-3279-48c2-98e7-84b71aca1188"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:55:23.992098 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.992026    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck2sp\" (UniqueName: \"kubernetes.io/projected/fd04cd64-3279-48c2-98e7-84b71aca1188-kube-api-access-ck2sp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:55:23.992098 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:23.992051    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd04cd64-3279-48c2-98e7-84b71aca1188-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:55:24.490063 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.490025    5510 generic.go:356] "Generic (PLEG): container finished" podID="fd04cd64-3279-48c2-98e7-84b71aca1188" containerID="0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8" exitCode=0
Nov 01 05:55:24.490507 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.490193    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8" event={"ID":"fd04cd64-3279-48c2-98e7-84b71aca1188","Type":"ContainerDied","Data":"0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8"}
Nov 01 05:55:24.490507 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.490261    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8" event={"ID":"fd04cd64-3279-48c2-98e7-84b71aca1188","Type":"ContainerDied","Data":"2686b9789414652dc3a462479768f0a1f5c5e87b679089736e2315a41f679adb"}
Nov 01 05:55:24.490507 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.490290    5510 scope.go:122] "RemoveContainer" containerID="0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8"
Nov 01 05:55:24.490507 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.490474    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6jlp8"
Nov 01 05:55:24.502396 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.502375    5510 scope.go:122] "RemoveContainer" containerID="0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8"
Nov 01 05:55:24.502645 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:55:24.502627    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8\": container with ID starting with 0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8 not found: ID does not exist" containerID="0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8"
Nov 01 05:55:24.502694 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.502657    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8"} err="failed to get container status \"0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8\": rpc error: code = NotFound desc = could not find container \"0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8\": container with ID starting with 0ce3e3a5c258f8ee7ea41f908fdd7af912cfec8177ca83a1d0ca900823cd0bd8 not found: ID does not exist"
Nov 01 05:55:24.655182 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.655146    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6jlp8"]
Nov 01 05:55:24.704519 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:24.704487    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6jlp8"]
Nov 01 05:55:25.234543 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:25.234508    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd04cd64-3279-48c2-98e7-84b71aca1188" path="/var/lib/kubelet/pods/fd04cd64-3279-48c2-98e7-84b71aca1188/volumes"
Nov 01 05:55:47.543573 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:55:47.543531    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:55:59.464992 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:55:59.464951    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:56:04.964271 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:04.964235    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"]
Nov 01 05:56:04.964635 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:04.964565    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fd04cd64-3279-48c2-98e7-84b71aca1188" containerName="manager"
Nov 01 05:56:04.964635 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:04.964578    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd04cd64-3279-48c2-98e7-84b71aca1188" containerName="manager"
Nov 01 05:56:04.964708 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:04.964660    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd04cd64-3279-48c2-98e7-84b71aca1188" containerName="manager"
Nov 01 05:56:04.968533 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:04.968517    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.038476 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.038449    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.038646 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.038580    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.038646 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.038608    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2hsc\" (UniqueName: \"kubernetes.io/projected/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-kube-api-access-b2hsc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.038765 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.038720    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.038822 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.038762    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.038885 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.038872    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.070314 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.070291    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 05:56:05.070513 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.070492    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 05:56:05.070585 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.070542    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 01 05:56:05.070585 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.070550    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4k6qc\""
Nov 01 05:56:05.071621 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.071604    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"]
Nov 01 05:56:05.139631 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.139597    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.139631 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.139631    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.139856 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.139651    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.139856 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.139768    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.139938 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.139918    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.139972 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.139948    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2hsc\" (UniqueName: \"kubernetes.io/projected/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-kube-api-access-b2hsc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.140115 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.140096    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.140181 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.140167    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.140240 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.140188    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.142297 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.142268    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.142577 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.142562    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.200889 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.200855    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2hsc\" (UniqueName: \"kubernetes.io/projected/0285b55a-dfd7-4f9c-a4c2-8acad8e49760-kube-api-access-b2hsc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4p5kf\" (UID: \"0285b55a-dfd7-4f9c-a4c2-8acad8e49760\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.281110 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.281019    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:05.651756 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.651685    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"]
Nov 01 05:56:05.654432 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:56:05.654403    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0285b55a_dfd7_4f9c_a4c2_8acad8e49760.slice/crio-1ca8546149b1c1146f79c009a5ab19e1ff197d27c1c9386408c350132bb3317e WatchSource:0}: Error finding container 1ca8546149b1c1146f79c009a5ab19e1ff197d27c1c9386408c350132bb3317e: Status 404 returned error can't find the container with id 1ca8546149b1c1146f79c009a5ab19e1ff197d27c1c9386408c350132bb3317e
Nov 01 05:56:05.671560 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:05.671528    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf" event={"ID":"0285b55a-dfd7-4f9c-a4c2-8acad8e49760","Type":"ContainerStarted","Data":"1ca8546149b1c1146f79c009a5ab19e1ff197d27c1c9386408c350132bb3317e"}
Nov 01 05:56:07.682214 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:07.682184    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf" event={"ID":"0285b55a-dfd7-4f9c-a4c2-8acad8e49760","Type":"ContainerStarted","Data":"06abf93850035b359e2a3a96a7046878d354240c8172f46c23ea6cc12251eb31"}
Nov 01 05:56:07.800244 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:07.800188    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf" podStartSLOduration=2.291400179 podStartE2EDuration="3.800175749s" totalImagesPullingTime="1.50877557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:07.798708569 +0000 UTC m=+604.676681983" watchObservedRunningTime="2026-11-01 05:56:07.800175749 +0000 UTC m=+604.678149159"
Nov 01 05:56:08.109471 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.109436    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"]
Nov 01 05:56:08.112945 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.112929    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.165101 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.165073    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 05:56:08.167396 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.167376    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.167447 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.167409    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.167490 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.167468    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.167634 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.167613    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d534516-5a40-48ba-8f09-6ac5cda95748-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.167700 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.167679    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.167752 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.167702    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvh7\" (UniqueName: \"kubernetes.io/projected/4d534516-5a40-48ba-8f09-6ac5cda95748-kube-api-access-qzvh7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.215367 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.215337    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"]
Nov 01 05:56:08.268913 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.268882    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.269051 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.268919    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.269051 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.268965    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.269051 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.268993    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d534516-5a40-48ba-8f09-6ac5cda95748-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.269177 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.269050    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.269177 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.269081    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvh7\" (UniqueName: \"kubernetes.io/projected/4d534516-5a40-48ba-8f09-6ac5cda95748-kube-api-access-qzvh7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.269340 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.269320    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.269380 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.269367    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.269424 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.269380    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.271472 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.271453    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d534516-5a40-48ba-8f09-6ac5cda95748-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.271618 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.271600    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d534516-5a40-48ba-8f09-6ac5cda95748-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.281492 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.281469    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:08.330190 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.330159    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvh7\" (UniqueName: \"kubernetes.io/projected/4d534516-5a40-48ba-8f09-6ac5cda95748-kube-api-access-qzvh7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9949x\" (UID: \"4d534516-5a40-48ba-8f09-6ac5cda95748\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.425087 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.425026    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:08.795526 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:08.795503    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"]
Nov 01 05:56:08.797919 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:56:08.797895    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d534516_5a40_48ba_8f09_6ac5cda95748.slice/crio-082db474794a79141bbb3eb8c4a43ba3e228937129aca8b442e7ca2650f3aa54 WatchSource:0}: Error finding container 082db474794a79141bbb3eb8c4a43ba3e228937129aca8b442e7ca2650f3aa54: Status 404 returned error can't find the container with id 082db474794a79141bbb3eb8c4a43ba3e228937129aca8b442e7ca2650f3aa54
Nov 01 05:56:09.693011 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:09.692923    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x" event={"ID":"4d534516-5a40-48ba-8f09-6ac5cda95748","Type":"ContainerStarted","Data":"8cd1d227328112edd032dc093c8899a3128907c60df9511f76567b727211b558"}
Nov 01 05:56:09.693011 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:09.692973    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x" event={"ID":"4d534516-5a40-48ba-8f09-6ac5cda95748","Type":"ContainerStarted","Data":"082db474794a79141bbb3eb8c4a43ba3e228937129aca8b442e7ca2650f3aa54"}
Nov 01 05:56:09.809770 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:09.809712    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x" podStartSLOduration=1.234865648 podStartE2EDuration="1.809698196s" totalImagesPullingTime="574.832548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:09.809309848 +0000 UTC m=+606.687283258" watchObservedRunningTime="2026-11-01 05:56:09.809698196 +0000 UTC m=+606.687671606"
Nov 01 05:56:10.307070 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:10.307037    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:56:10.426100 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:10.426063    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:15.281648 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:15.281612    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:15.293994 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:15.293971    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:15.732694 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:15.732662    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4p5kf"
Nov 01 05:56:16.545441 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.545396    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"]
Nov 01 05:56:16.548956 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.548940    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.634579 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.634552    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 05:56:16.639948 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.639929    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.640031 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.639980    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.640031 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.640007    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ade8493-4326-41f7-8be6-202a3f237998-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.640119 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.640040    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7vs\" (UniqueName: \"kubernetes.io/projected/9ade8493-4326-41f7-8be6-202a3f237998-kube-api-access-gp7vs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.640162 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.640147    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.640196 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.640175    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.651643 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.651618    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"]
Nov 01 05:56:16.741520 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.741492    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.741674 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.741526    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ade8493-4326-41f7-8be6-202a3f237998-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.741674 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.741549    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7vs\" (UniqueName: \"kubernetes.io/projected/9ade8493-4326-41f7-8be6-202a3f237998-kube-api-access-gp7vs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.741674 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.741584    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.741674 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.741609    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.741904 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.741698    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.741964 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.741923    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.742017 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.741999    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.742102 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.742085    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.743952 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.743930    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ade8493-4326-41f7-8be6-202a3f237998-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.744137 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.744122    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ade8493-4326-41f7-8be6-202a3f237998-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.814781 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.814725    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7vs\" (UniqueName: \"kubernetes.io/projected/9ade8493-4326-41f7-8be6-202a3f237998-kube-api-access-gp7vs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv\" (UID: \"9ade8493-4326-41f7-8be6-202a3f237998\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:16.861505 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:16.861473    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:17.243424 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:56:17.243396    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ade8493_4326_41f7_8be6_202a3f237998.slice/crio-124218200a54462ba4aeb8ddae10870ae86d32c228a325469512c4f0cd82db15 WatchSource:0}: Error finding container 124218200a54462ba4aeb8ddae10870ae86d32c228a325469512c4f0cd82db15: Status 404 returned error can't find the container with id 124218200a54462ba4aeb8ddae10870ae86d32c228a325469512c4f0cd82db15
Nov 01 05:56:17.243626 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:17.243602    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"]
Nov 01 05:56:17.544844 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:56:17.544740    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:56:17.731945 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:17.731910    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv" event={"ID":"9ade8493-4326-41f7-8be6-202a3f237998","Type":"ContainerStarted","Data":"124218200a54462ba4aeb8ddae10870ae86d32c228a325469512c4f0cd82db15"}
Nov 01 05:56:18.425894 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:18.425856    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:18.437756 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:18.437733    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:18.740039 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:18.739996    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv" event={"ID":"9ade8493-4326-41f7-8be6-202a3f237998","Type":"ContainerStarted","Data":"039f255ad9c7872f69e4115e3e060ca3f3a0f939d2f777ae5989e43d9e84ba2f"}
Nov 01 05:56:18.751879 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:18.751856    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9949x"
Nov 01 05:56:18.858115 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:18.858060    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv" podStartSLOduration=2.256418005 podStartE2EDuration="2.858044205s" totalImagesPullingTime="601.6262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:18.857643461 +0000 UTC m=+615.735616873" watchObservedRunningTime="2026-11-01 05:56:18.858044205 +0000 UTC m=+615.736017614"
Nov 01 05:56:18.861897 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:18.861874    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:24.213032 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.212991    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"]
Nov 01 05:56:24.218920 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.218895    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.308686 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.308659    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.308881 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.308732    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7hzx\" (UniqueName: \"kubernetes.io/projected/b96ce760-802d-4382-9c01-8d8a588a9810-kube-api-access-h7hzx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.308969 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.308914    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.309032 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.308976    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.309100 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.309039    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96ce760-802d-4382-9c01-8d8a588a9810-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.309100 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.309073    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.322598 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.322576    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 05:56:24.323795 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.323778    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"]
Nov 01 05:56:24.410429 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.410396    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7hzx\" (UniqueName: \"kubernetes.io/projected/b96ce760-802d-4382-9c01-8d8a588a9810-kube-api-access-h7hzx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.410583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.410434    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.410583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.410476    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.410583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.410507    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96ce760-802d-4382-9c01-8d8a588a9810-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.410583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.410535    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.410583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.410578    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.411032 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.411000    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.411032 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.411030    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.411169 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.411073    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.412966 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.412944    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96ce760-802d-4382-9c01-8d8a588a9810-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.413100 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.413082    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96ce760-802d-4382-9c01-8d8a588a9810-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.471456 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.471387    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7hzx\" (UniqueName: \"kubernetes.io/projected/b96ce760-802d-4382-9c01-8d8a588a9810-kube-api-access-h7hzx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp\" (UID: \"b96ce760-802d-4382-9c01-8d8a588a9810\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.532162 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.532131    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:24.902239 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:24.902210    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"]
Nov 01 05:56:24.904556 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:56:24.904530    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ce760_802d_4382_9c01_8d8a588a9810.slice/crio-8b6830a980fa08659be3ea2e5a4b2232a1e34a3acaaa1815730659947688a942 WatchSource:0}: Error finding container 8b6830a980fa08659be3ea2e5a4b2232a1e34a3acaaa1815730659947688a942: Status 404 returned error can't find the container with id 8b6830a980fa08659be3ea2e5a4b2232a1e34a3acaaa1815730659947688a942
Nov 01 05:56:25.774265 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:25.774170    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp" event={"ID":"b96ce760-802d-4382-9c01-8d8a588a9810","Type":"ContainerStarted","Data":"716b7c4c730be5edb1c5069c13202faf78f59a8b92482c29afdfdb4cd006951f"}
Nov 01 05:56:25.774265 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:25.774230    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp" event={"ID":"b96ce760-802d-4382-9c01-8d8a588a9810","Type":"ContainerStarted","Data":"8b6830a980fa08659be3ea2e5a4b2232a1e34a3acaaa1815730659947688a942"}
Nov 01 05:56:25.890967 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:25.890915    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp" podStartSLOduration=1.318921348 podStartE2EDuration="1.890900786s" totalImagesPullingTime="571.979438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:25.890106311 +0000 UTC m=+622.768079722" watchObservedRunningTime="2026-11-01 05:56:25.890900786 +0000 UTC m=+622.768874225"
Nov 01 05:56:26.532774 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:26.532735    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:26.862238 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:26.862160    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:26.874352 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:26.874332    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:27.794357 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:27.794329    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv"
Nov 01 05:56:28.091613 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:28.091526    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:56:34.532737 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:34.532702    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:34.545229 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:34.545200    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:34.827073 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:34.826994    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp"
Nov 01 05:56:40.912791 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:40.912758    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:56:47.543738 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:56:47.543647    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:56:53.274019 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.273990    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"]
Nov 01 05:56:53.277379 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.277359    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.329248 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.329218    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vqwlh\""
Nov 01 05:56:53.390410 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.390372    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"]
Nov 01 05:56:53.469359 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469333    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.469501 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469366    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.469501 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469385    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4ht\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-kube-api-access-zg4ht\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.469582 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469528    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.469615 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469583    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.469662 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469628    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.469698 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469661    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.469698 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469678    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e03b3-0dbf-4e33-ac73-49c364611072-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.469759 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.469706    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.570593 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570519    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.570593 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570570    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.570593 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570588    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.570899 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570618    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.570899 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570641    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e03b3-0dbf-4e33-ac73-49c364611072-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.570899 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570681    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.570899 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570712    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.571114 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570909    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.571114 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.570941    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4ht\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-kube-api-access-zg4ht\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.571114 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.571020    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.571114 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.571083    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.571256 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.571136    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.571256 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.571137    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.571743 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.571720    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e03b3-0dbf-4e33-ac73-49c364611072-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.573162 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.573145    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.573597 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.573582    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.631347 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.631309    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.631465 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.631380    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4ht\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-kube-api-access-zg4ht\") pod \"maas-default-gateway-openshift-default-855f9574cf-r6pgr\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:53.890448 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:53.890372    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:56:54.316093 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:54.316052    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"]
Nov 01 05:56:54.318454 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:56:54.318429    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6e03b3_0dbf_4e33_ac73_49c364611072.slice/crio-99bd721ba5690fabe1b166eeebe17fbbb1e39c2651350718a175548b07898528 WatchSource:0}: Error finding container 99bd721ba5690fabe1b166eeebe17fbbb1e39c2651350718a175548b07898528: Status 404 returned error can't find the container with id 99bd721ba5690fabe1b166eeebe17fbbb1e39c2651350718a175548b07898528
Nov 01 05:56:54.320358 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:54.320341    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:56:54.910221 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:54.910155    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" event={"ID":"bf6e03b3-0dbf-4e33-ac73-49c364611072","Type":"ContainerStarted","Data":"99bd721ba5690fabe1b166eeebe17fbbb1e39c2651350718a175548b07898528"}
Nov 01 05:56:59.072635 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:59.072605    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:56:59.072908 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:59.072690    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:56:59.072908 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:59.072720    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:56:59.934387 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:56:59.934345    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" event={"ID":"bf6e03b3-0dbf-4e33-ac73-49c364611072","Type":"ContainerStarted","Data":"11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c"}
Nov 01 05:57:00.160101 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:00.160051    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" podStartSLOduration=2.408772028 podStartE2EDuration="7.160027047s" totalImagesPullingTime="4.751255019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:57:00.109791469 +0000 UTC m=+656.987764882" watchObservedRunningTime="2026-11-01 05:57:00.160027047 +0000 UTC m=+657.038000457"
Nov 01 05:57:00.893118 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:00.893087    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:57:00.895376 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:00.895348    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-r6pgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:57:00.895511 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:00.895410    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 05:57:01.894175 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:01.894139    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-r6pgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:57:01.894598 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:01.894215    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 05:57:02.311448 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.311408    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-696f7c4d5c-6px8t"]
Nov 01 05:57:02.311694 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.311647    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-696f7c4d5c-6px8t" podUID="d211e9b9-226b-422c-9408-7a1690a1064f" containerName="authorino" containerID="cri-o://9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7" gracePeriod=30
Nov 01 05:57:02.603126 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.603099    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:57:02.657112 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.657086    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d211e9b9-226b-422c-9408-7a1690a1064f-kube-api-access-2465j\" (UniqueName: \"kubernetes.io/projected/d211e9b9-226b-422c-9408-7a1690a1064f-kube-api-access-2465j\") pod \"d211e9b9-226b-422c-9408-7a1690a1064f\" (UID: \"d211e9b9-226b-422c-9408-7a1690a1064f\") "
Nov 01 05:57:02.657260 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.657123    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d211e9b9-226b-422c-9408-7a1690a1064f-tls-cert\" (UniqueName: \"kubernetes.io/secret/d211e9b9-226b-422c-9408-7a1690a1064f-tls-cert\") pod \"d211e9b9-226b-422c-9408-7a1690a1064f\" (UID: \"d211e9b9-226b-422c-9408-7a1690a1064f\") "
Nov 01 05:57:02.659377 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.659341    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d211e9b9-226b-422c-9408-7a1690a1064f-kube-api-access-2465j" pod "d211e9b9-226b-422c-9408-7a1690a1064f" (UID: "d211e9b9-226b-422c-9408-7a1690a1064f"). InnerVolumeSpecName "kube-api-access-2465j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:57:02.667366 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.667343    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d211e9b9-226b-422c-9408-7a1690a1064f-tls-cert" pod "d211e9b9-226b-422c-9408-7a1690a1064f" (UID: "d211e9b9-226b-422c-9408-7a1690a1064f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:57:02.757944 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.757917    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2465j\" (UniqueName: \"kubernetes.io/projected/d211e9b9-226b-422c-9408-7a1690a1064f-kube-api-access-2465j\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:02.757944 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.757943    5510 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d211e9b9-226b-422c-9408-7a1690a1064f-tls-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:02.893765 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.893208    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-r6pgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:57:02.893765 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.893285    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 05:57:02.949467 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.949435    5510 generic.go:356] "Generic (PLEG): container finished" podID="d211e9b9-226b-422c-9408-7a1690a1064f" containerID="9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7" exitCode=0
Nov 01 05:57:02.949914 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.949485    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-696f7c4d5c-6px8t"
Nov 01 05:57:02.949914 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.949499    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-696f7c4d5c-6px8t" event={"ID":"d211e9b9-226b-422c-9408-7a1690a1064f","Type":"ContainerDied","Data":"9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7"}
Nov 01 05:57:02.949914 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.949535    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-696f7c4d5c-6px8t" event={"ID":"d211e9b9-226b-422c-9408-7a1690a1064f","Type":"ContainerDied","Data":"ab52fdf779c6cc7ee9f56ff14af3f6bb1ea74c77b0615f3121c7e2fd0f16d262"}
Nov 01 05:57:02.949914 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.949558    5510 scope.go:122] "RemoveContainer" containerID="9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7"
Nov 01 05:57:02.958938 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.958923    5510 scope.go:122] "RemoveContainer" containerID="9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7"
Nov 01 05:57:02.959206 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:57:02.959185    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7\": container with ID starting with 9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7 not found: ID does not exist" containerID="9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7"
Nov 01 05:57:02.959256 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:02.959216    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7"} err="failed to get container status \"9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7\": rpc error: code = NotFound desc = could not find container \"9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7\": container with ID starting with 9670eb041af392057a9c31f6f5caf683bea830c7584bec3a00cf610ab84494b7 not found: ID does not exist"
Nov 01 05:57:03.111929 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:03.111898    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-696f7c4d5c-6px8t"]
Nov 01 05:57:03.114024 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:03.114002    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-696f7c4d5c-6px8t"]
Nov 01 05:57:03.233686 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:03.233659    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d211e9b9-226b-422c-9408-7a1690a1064f" path="/var/lib/kubelet/pods/d211e9b9-226b-422c-9408-7a1690a1064f/volumes"
Nov 01 05:57:03.893249 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:03.893217    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-r6pgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:57:03.893531 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:03.893505    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 05:57:04.894201 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:04.894167    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-r6pgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:57:04.894568 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:04.894221    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 05:57:05.891799 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:05.891760    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-r6pgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:57:05.891987 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:05.891817    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 05:57:07.585008 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:07.584974    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:57:07.585878 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:07.585525    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:57:07.586222 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:07.586204    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 05:57:17.547847 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:57:17.547780    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:57:47.544725 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:57:47.544684    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:57:50.752106 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.752067    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"]
Nov 01 05:57:50.752504 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.752462    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d211e9b9-226b-422c-9408-7a1690a1064f" containerName="authorino"
Nov 01 05:57:50.752504 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.752473    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d211e9b9-226b-422c-9408-7a1690a1064f" containerName="authorino"
Nov 01 05:57:50.752583 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.752545    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d211e9b9-226b-422c-9408-7a1690a1064f" containerName="authorino"
Nov 01 05:57:50.755414 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.755399    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795563 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795526    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795700 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795581    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkd75\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-kube-api-access-gkd75\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795700 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795639    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795700 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795671    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795700 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795684    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795870 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795713    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795870 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795806    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795956 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795885    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.795956 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.795906    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.807761 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.807737    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9c8fc5-openshift-default-dockercfg-6mn84\""
Nov 01 05:57:50.858462 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.858438    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"]
Nov 01 05:57:50.900635 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.900598    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.900849 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.900665    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkd75\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-kube-api-access-gkd75\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.900988 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.900901    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.900988 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.900951    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.900988 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.900976    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.901158 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.901014    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.901158 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.901046    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.901158 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.901086    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.901158 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.901114    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.901350 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.901154    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.901995 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.901949    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.902343 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.902314    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.902464 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.902446    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.902525 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.902481    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.903775 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.903752    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.904278 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.904254    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.962014 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.961980    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:50.962165 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:50.962122    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkd75\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-kube-api-access-gkd75\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:51.067810 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:51.067731    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:51.502589 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:51.502567    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"]
Nov 01 05:57:51.505253 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:57:51.505225    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efa1d66_775f_4ba4_ae2e_658da02e7ad0.slice/crio-a065c4a0e413551690144f87a52fdfa5f8e126a04b4c847f1dfd256ec64f8b64 WatchSource:0}: Error finding container a065c4a0e413551690144f87a52fdfa5f8e126a04b4c847f1dfd256ec64f8b64: Status 404 returned error can't find the container with id a065c4a0e413551690144f87a52fdfa5f8e126a04b4c847f1dfd256ec64f8b64
Nov 01 05:57:51.507385 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:51.507353    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:57:51.507448 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:51.507418    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:57:51.507489 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:51.507447    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:57:52.166333 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:52.166299    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" event={"ID":"8efa1d66-775f-4ba4-ae2e-658da02e7ad0","Type":"ContainerStarted","Data":"be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f"}
Nov 01 05:57:52.166727 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:52.166346    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" event={"ID":"8efa1d66-775f-4ba4-ae2e-658da02e7ad0","Type":"ContainerStarted","Data":"a065c4a0e413551690144f87a52fdfa5f8e126a04b4c847f1dfd256ec64f8b64"}
Nov 01 05:57:52.286943 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:52.286893    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podStartSLOduration=2.286878805 podStartE2EDuration="2.286878805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:57:52.28598107 +0000 UTC m=+709.163954483" watchObservedRunningTime="2026-11-01 05:57:52.286878805 +0000 UTC m=+709.164852212"
Nov 01 05:57:53.068154 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:53.068113    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:53.073072 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:53.073048    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:53.171114 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:53.171084    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:53.171977 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:53.171960    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:57:58.658303 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.658274    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"]
Nov 01 05:57:58.661844 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.661809    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:57:58.713748 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.713721    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f73e89-dockercfg-mnxxh\""
Nov 01 05:57:58.763119 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.763076    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f73e89\""
Nov 01 05:57:58.764638 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.764604    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"]
Nov 01 05:57:58.775977 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.775948    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrnj\" (UniqueName: \"kubernetes.io/projected/5e7eaa36-4938-4d11-8db2-304a18f0de38-kube-api-access-clrnj\") pod \"payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh\" (UID: \"5e7eaa36-4938-4d11-8db2-304a18f0de38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:57:58.776112 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.776093    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7eaa36-4938-4d11-8db2-304a18f0de38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh\" (UID: \"5e7eaa36-4938-4d11-8db2-304a18f0de38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:57:58.813946 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.813921    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"]
Nov 01 05:57:58.817437 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.817421    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:57:58.871508 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.871484    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"]
Nov 01 05:57:58.877319 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.877296    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7eaa36-4938-4d11-8db2-304a18f0de38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh\" (UID: \"5e7eaa36-4938-4d11-8db2-304a18f0de38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:57:58.877448 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.877370    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkgs\" (UniqueName: \"kubernetes.io/projected/8d987f61-f6ba-4950-9ba2-7023096db483-kube-api-access-2nkgs\") pod \"payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp\" (UID: \"8d987f61-f6ba-4950-9ba2-7023096db483\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:57:58.877448 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.877397    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrnj\" (UniqueName: \"kubernetes.io/projected/5e7eaa36-4938-4d11-8db2-304a18f0de38-kube-api-access-clrnj\") pod \"payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh\" (UID: \"5e7eaa36-4938-4d11-8db2-304a18f0de38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:57:58.877569 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.877502    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d987f61-f6ba-4950-9ba2-7023096db483-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp\" (UID: \"8d987f61-f6ba-4950-9ba2-7023096db483\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:57:58.878035 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.878014    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7eaa36-4938-4d11-8db2-304a18f0de38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh\" (UID: \"5e7eaa36-4938-4d11-8db2-304a18f0de38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:57:58.945066 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.945008    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrnj\" (UniqueName: \"kubernetes.io/projected/5e7eaa36-4938-4d11-8db2-304a18f0de38-kube-api-access-clrnj\") pod \"payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh\" (UID: \"5e7eaa36-4938-4d11-8db2-304a18f0de38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:57:58.974821 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.974795    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:57:58.978052 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.978031    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d987f61-f6ba-4950-9ba2-7023096db483-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp\" (UID: \"8d987f61-f6ba-4950-9ba2-7023096db483\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:57:58.978179 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.978161    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkgs\" (UniqueName: \"kubernetes.io/projected/8d987f61-f6ba-4950-9ba2-7023096db483-kube-api-access-2nkgs\") pod \"payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp\" (UID: \"8d987f61-f6ba-4950-9ba2-7023096db483\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:57:58.978654 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:58.978632    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d987f61-f6ba-4950-9ba2-7023096db483-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp\" (UID: \"8d987f61-f6ba-4950-9ba2-7023096db483\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:57:59.046485 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:59.046452    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkgs\" (UniqueName: \"kubernetes.io/projected/8d987f61-f6ba-4950-9ba2-7023096db483-kube-api-access-2nkgs\") pod \"payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp\" (UID: \"8d987f61-f6ba-4950-9ba2-7023096db483\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:57:59.131605 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:59.131574    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:57:59.348577 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:59.348553    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"]
Nov 01 05:57:59.353221 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:57:59.353194    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7eaa36_4938_4d11_8db2_304a18f0de38.slice/crio-6ee8c574dc8007c89cd86a4e4b26af408ef9abfc834f481259f4fc9b6a542f35 WatchSource:0}: Error finding container 6ee8c574dc8007c89cd86a4e4b26af408ef9abfc834f481259f4fc9b6a542f35: Status 404 returned error can't find the container with id 6ee8c574dc8007c89cd86a4e4b26af408ef9abfc834f481259f4fc9b6a542f35
Nov 01 05:57:59.504429 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:57:59.504409    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"]
Nov 01 05:57:59.506579 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:57:59.506558    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d987f61_f6ba_4950_9ba2_7023096db483.slice/crio-d775d30c6111c6d6798e0f891d385a506054f1c878f9d740b1a3c32282a758cd WatchSource:0}: Error finding container d775d30c6111c6d6798e0f891d385a506054f1c878f9d740b1a3c32282a758cd: Status 404 returned error can't find the container with id d775d30c6111c6d6798e0f891d385a506054f1c878f9d740b1a3c32282a758cd
Nov 01 05:58:00.207105 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.207070    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp" event={"ID":"8d987f61-f6ba-4950-9ba2-7023096db483","Type":"ContainerStarted","Data":"25db57e67ed6a9191c1500bccc4c6074e099aed23e5d5209c9f141764fdad3da"}
Nov 01 05:58:00.207594 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.207139    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:58:00.207594 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.207163    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp" event={"ID":"8d987f61-f6ba-4950-9ba2-7023096db483","Type":"ContainerStarted","Data":"d775d30c6111c6d6798e0f891d385a506054f1c878f9d740b1a3c32282a758cd"}
Nov 01 05:58:00.208761 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.208735    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh" event={"ID":"5e7eaa36-4938-4d11-8db2-304a18f0de38","Type":"ContainerStarted","Data":"0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f"}
Nov 01 05:58:00.208917 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.208778    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh" event={"ID":"5e7eaa36-4938-4d11-8db2-304a18f0de38","Type":"ContainerStarted","Data":"6ee8c574dc8007c89cd86a4e4b26af408ef9abfc834f481259f4fc9b6a542f35"}
Nov 01 05:58:00.208917 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.208807    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:58:00.323580 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.323514    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp" podStartSLOduration=2.323487496 podStartE2EDuration="2.323487496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:00.322132143 +0000 UTC m=+717.200105555" watchObservedRunningTime="2026-11-01 05:58:00.323487496 +0000 UTC m=+717.201460907"
Nov 01 05:58:00.438788 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.438729    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh" podStartSLOduration=2.438711715 podStartE2EDuration="2.438711715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:00.437910682 +0000 UTC m=+717.315884093" watchObservedRunningTime="2026-11-01 05:58:00.438711715 +0000 UTC m=+717.316685125"
Nov 01 05:58:00.767056 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.767024    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"]
Nov 01 05:58:00.770721 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.770699    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.822857 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.822813    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-14a6e4-openshift-default-dockercfg-vxhc4\""
Nov 01 05:58:00.832062 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.831844    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"]
Nov 01 05:58:00.896228 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896197    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mch2q\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-kube-api-access-mch2q\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.896369 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896247    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.896369 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896336    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75de03a3-8add-411c-a293-9487c9b218ad-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.896467 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896386    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75de03a3-8add-411c-a293-9487c9b218ad-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.896467 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896441    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.896537 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896466    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.896537 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896514    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.896609 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896566    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.896609 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.896591    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997424 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997389    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997606 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997432    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75de03a3-8add-411c-a293-9487c9b218ad-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997606 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997450    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75de03a3-8add-411c-a293-9487c9b218ad-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997606 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997470    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997608    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997729    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997766    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997964 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997803    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997964 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997881    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mch2q\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-kube-api-access-mch2q\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.997964 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.997933    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.998223 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.998197    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.998349 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.998297    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.998349 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.998305    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75de03a3-8add-411c-a293-9487c9b218ad-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:00.998553 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:00.998460    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:01.000226 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.000201    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:01.001131 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.001110    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75de03a3-8add-411c-a293-9487c9b218ad-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:01.058806 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.058738    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mch2q\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-kube-api-access-mch2q\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:01.058978 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.058882    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:01.083959 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.083934    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:01.511483 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.511459    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"]
Nov 01 05:58:01.513791 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:58:01.513762    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75de03a3_8add_411c_a293_9487c9b218ad.slice/crio-da90a739dd842f82792e369116924ecbe024f405a64b871acd6df3dae99d78c3 WatchSource:0}: Error finding container da90a739dd842f82792e369116924ecbe024f405a64b871acd6df3dae99d78c3: Status 404 returned error can't find the container with id da90a739dd842f82792e369116924ecbe024f405a64b871acd6df3dae99d78c3
Nov 01 05:58:01.515998 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.515962    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:01.516104 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.516042    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:01.516104 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:01.516083    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:02.219719 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:02.219681    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" event={"ID":"75de03a3-8add-411c-a293-9487c9b218ad","Type":"ContainerStarted","Data":"a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac"}
Nov 01 05:58:02.219942 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:02.219749    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" event={"ID":"75de03a3-8add-411c-a293-9487c9b218ad","Type":"ContainerStarted","Data":"da90a739dd842f82792e369116924ecbe024f405a64b871acd6df3dae99d78c3"}
Nov 01 05:58:02.340180 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:02.340131    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" podStartSLOduration=2.34011724 podStartE2EDuration="2.34011724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:02.338118498 +0000 UTC m=+719.216091909" watchObservedRunningTime="2026-11-01 05:58:02.34011724 +0000 UTC m=+719.218090650"
Nov 01 05:58:03.084751 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.084717    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:03.096484 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.096449    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:03.224881 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.224820    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:03.225654 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.225627    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:03.414706 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.414633    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"]
Nov 01 05:58:03.419028 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.419008    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:03.474700 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.474676    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9c8fc5-dockercfg-2phfz\""
Nov 01 05:58:03.474895 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.474785    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9c8fc5\""
Nov 01 05:58:03.476795 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.476773    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"]
Nov 01 05:58:03.523488 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.523458    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxklq\" (UniqueName: \"kubernetes.io/projected/4e157229-ba78-4611-91b8-270c29c0dcdc-kube-api-access-wxklq\") pod \"payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds\" (UID: \"4e157229-ba78-4611-91b8-270c29c0dcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:03.523651 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.523516    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e157229-ba78-4611-91b8-270c29c0dcdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds\" (UID: \"4e157229-ba78-4611-91b8-270c29c0dcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:03.568573 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.568539    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"]
Nov 01 05:58:03.572267 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.572240    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:03.624853 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.624802    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357950be-1cc3-4216-95a9-376698aebd91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m\" (UID: \"357950be-1cc3-4216-95a9-376698aebd91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:03.625009 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.624908    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e157229-ba78-4611-91b8-270c29c0dcdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds\" (UID: \"4e157229-ba78-4611-91b8-270c29c0dcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:03.625009 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.625001    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27ft\" (UniqueName: \"kubernetes.io/projected/357950be-1cc3-4216-95a9-376698aebd91-kube-api-access-k27ft\") pod \"payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m\" (UID: \"357950be-1cc3-4216-95a9-376698aebd91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:03.625168 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.625144    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxklq\" (UniqueName: \"kubernetes.io/projected/4e157229-ba78-4611-91b8-270c29c0dcdc-kube-api-access-wxklq\") pod \"payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds\" (UID: \"4e157229-ba78-4611-91b8-270c29c0dcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:03.625668 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.625616    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e157229-ba78-4611-91b8-270c29c0dcdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds\" (UID: \"4e157229-ba78-4611-91b8-270c29c0dcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:03.628171 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.628148    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"]
Nov 01 05:58:03.685678 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.685618    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxklq\" (UniqueName: \"kubernetes.io/projected/4e157229-ba78-4611-91b8-270c29c0dcdc-kube-api-access-wxklq\") pod \"payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds\" (UID: \"4e157229-ba78-4611-91b8-270c29c0dcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:03.725753 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.725728    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27ft\" (UniqueName: \"kubernetes.io/projected/357950be-1cc3-4216-95a9-376698aebd91-kube-api-access-k27ft\") pod \"payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m\" (UID: \"357950be-1cc3-4216-95a9-376698aebd91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:03.725934 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.725854    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357950be-1cc3-4216-95a9-376698aebd91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m\" (UID: \"357950be-1cc3-4216-95a9-376698aebd91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:03.726477 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.726460    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357950be-1cc3-4216-95a9-376698aebd91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m\" (UID: \"357950be-1cc3-4216-95a9-376698aebd91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:03.731648 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.731624    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:03.790408 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.790380    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27ft\" (UniqueName: \"kubernetes.io/projected/357950be-1cc3-4216-95a9-376698aebd91-kube-api-access-k27ft\") pod \"payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m\" (UID: \"357950be-1cc3-4216-95a9-376698aebd91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:03.888497 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:03.888450    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:04.100397 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:04.100376    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"]
Nov 01 05:58:04.103573 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:58:04.103549    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e157229_ba78_4611_91b8_270c29c0dcdc.slice/crio-ed18e338121c9505597552793452e12c376a2ccb91700951f585265f4df540aa WatchSource:0}: Error finding container ed18e338121c9505597552793452e12c376a2ccb91700951f585265f4df540aa: Status 404 returned error can't find the container with id ed18e338121c9505597552793452e12c376a2ccb91700951f585265f4df540aa
Nov 01 05:58:04.230539 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:04.230500    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds" event={"ID":"4e157229-ba78-4611-91b8-270c29c0dcdc","Type":"ContainerStarted","Data":"84c5f81542394ffdf83fe73683db492ace80ca8db8cdeab51c16652ac8e332ea"}
Nov 01 05:58:04.230712 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:04.230555    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds" event={"ID":"4e157229-ba78-4611-91b8-270c29c0dcdc","Type":"ContainerStarted","Data":"ed18e338121c9505597552793452e12c376a2ccb91700951f585265f4df540aa"}
Nov 01 05:58:04.230712 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:04.230606    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:04.258964 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:04.258942    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"]
Nov 01 05:58:04.261257 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:58:04.261230    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357950be_1cc3_4216_95a9_376698aebd91.slice/crio-cf1bc22cc50a5601c61daaad83c39a73372afee7b90c61aacb521ee8d8adf0c7 WatchSource:0}: Error finding container cf1bc22cc50a5601c61daaad83c39a73372afee7b90c61aacb521ee8d8adf0c7: Status 404 returned error can't find the container with id cf1bc22cc50a5601c61daaad83c39a73372afee7b90c61aacb521ee8d8adf0c7
Nov 01 05:58:04.345856 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:04.345793    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds" podStartSLOduration=1.345780115 podStartE2EDuration="1.345780115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:04.344528543 +0000 UTC m=+721.222501954" watchObservedRunningTime="2026-11-01 05:58:04.345780115 +0000 UTC m=+721.223753524"
Nov 01 05:58:05.237205 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:05.237171    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m" event={"ID":"357950be-1cc3-4216-95a9-376698aebd91","Type":"ContainerStarted","Data":"2e34985c8f92d54b7d23fd784cbb2a588223e01b1001e96b3c41266fadca465a"}
Nov 01 05:58:05.237640 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:05.237229    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m" event={"ID":"357950be-1cc3-4216-95a9-376698aebd91","Type":"ContainerStarted","Data":"cf1bc22cc50a5601c61daaad83c39a73372afee7b90c61aacb521ee8d8adf0c7"}
Nov 01 05:58:05.237640 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:05.237344    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:05.352447 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:05.352397    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m" podStartSLOduration=2.352384792 podStartE2EDuration="2.352384792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:05.352053921 +0000 UTC m=+722.230027328" watchObservedRunningTime="2026-11-01 05:58:05.352384792 +0000 UTC m=+722.230358202"
Nov 01 05:58:11.215754 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:11.215725    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 05:58:11.216174 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:11.215791    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 05:58:14.200298 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:14.200222    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:58:15.240080 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:15.240056    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 05:58:16.245866 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:16.245822    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 05:58:17.544764 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:58:17.544726    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:58:18.858501 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:18.858465    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"]
Nov 01 05:58:18.862056 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:18.862041    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:18.968760 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:18.968732    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-14a6e4\""
Nov 01 05:58:18.969109 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:18.969039    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-14a6e4-dockercfg-8vth7\""
Nov 01 05:58:18.971845 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:18.971798    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"]
Nov 01 05:58:18.999068 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:18.998925    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ed2f5-f019-4457-a3f9-b99da85c104f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq\" (UID: \"a82ed2f5-f019-4457-a3f9-b99da85c104f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:18.999068 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:18.998986    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbp8g\" (UniqueName: \"kubernetes.io/projected/a82ed2f5-f019-4457-a3f9-b99da85c104f-kube-api-access-hbp8g\") pod \"payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq\" (UID: \"a82ed2f5-f019-4457-a3f9-b99da85c104f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:19.019928 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.019893    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"]
Nov 01 05:58:19.023340 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.023326    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:19.064822 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.064798    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"]
Nov 01 05:58:19.099852 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.099699    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbp8g\" (UniqueName: \"kubernetes.io/projected/a82ed2f5-f019-4457-a3f9-b99da85c104f-kube-api-access-hbp8g\") pod \"payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq\" (UID: \"a82ed2f5-f019-4457-a3f9-b99da85c104f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:19.099852 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.099795    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7176c24-60a4-4250-822f-84fd0b79b881-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q\" (UID: \"e7176c24-60a4-4250-822f-84fd0b79b881\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:19.100094 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.099892    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ed2f5-f019-4457-a3f9-b99da85c104f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq\" (UID: \"a82ed2f5-f019-4457-a3f9-b99da85c104f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:19.100094 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.099944    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95fc\" (UniqueName: \"kubernetes.io/projected/e7176c24-60a4-4250-822f-84fd0b79b881-kube-api-access-t95fc\") pod \"payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q\" (UID: \"e7176c24-60a4-4250-822f-84fd0b79b881\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:19.100903 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.100879    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ed2f5-f019-4457-a3f9-b99da85c104f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq\" (UID: \"a82ed2f5-f019-4457-a3f9-b99da85c104f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:19.161451 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.161384    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbp8g\" (UniqueName: \"kubernetes.io/projected/a82ed2f5-f019-4457-a3f9-b99da85c104f-kube-api-access-hbp8g\") pod \"payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq\" (UID: \"a82ed2f5-f019-4457-a3f9-b99da85c104f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:19.198172 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.198138    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:19.200565 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.200540    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95fc\" (UniqueName: \"kubernetes.io/projected/e7176c24-60a4-4250-822f-84fd0b79b881-kube-api-access-t95fc\") pod \"payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q\" (UID: \"e7176c24-60a4-4250-822f-84fd0b79b881\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:19.200657 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.200614    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7176c24-60a4-4250-822f-84fd0b79b881-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q\" (UID: \"e7176c24-60a4-4250-822f-84fd0b79b881\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:19.201378 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.201310    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7176c24-60a4-4250-822f-84fd0b79b881-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q\" (UID: \"e7176c24-60a4-4250-822f-84fd0b79b881\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:19.261611 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.261540    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95fc\" (UniqueName: \"kubernetes.io/projected/e7176c24-60a4-4250-822f-84fd0b79b881-kube-api-access-t95fc\") pod \"payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q\" (UID: \"e7176c24-60a4-4250-822f-84fd0b79b881\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:19.337103 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.337071    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:19.589855 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.571377    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"]
Nov 01 05:58:19.594355 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:58:19.594325    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82ed2f5_f019_4457_a3f9_b99da85c104f.slice/crio-dc45f9d23cb32b99391e4d6b794d737fd83bd722e2829340199c8abfc2a8c515 WatchSource:0}: Error finding container dc45f9d23cb32b99391e4d6b794d737fd83bd722e2829340199c8abfc2a8c515: Status 404 returned error can't find the container with id dc45f9d23cb32b99391e4d6b794d737fd83bd722e2829340199c8abfc2a8c515
Nov 01 05:58:19.732287 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:19.732259    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"]
Nov 01 05:58:19.735241 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:58:19.735212    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7176c24_60a4_4250_822f_84fd0b79b881.slice/crio-ed7d6a84d1d1d8842fd473ffe32da59b0e7ee062dc03d932868a95e01a9aac4e WatchSource:0}: Error finding container ed7d6a84d1d1d8842fd473ffe32da59b0e7ee062dc03d932868a95e01a9aac4e: Status 404 returned error can't find the container with id ed7d6a84d1d1d8842fd473ffe32da59b0e7ee062dc03d932868a95e01a9aac4e
Nov 01 05:58:20.314793 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:20.314756    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q" event={"ID":"e7176c24-60a4-4250-822f-84fd0b79b881","Type":"ContainerStarted","Data":"b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3"}
Nov 01 05:58:20.315256 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:20.314935    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:20.315256 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:20.314984    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q" event={"ID":"e7176c24-60a4-4250-822f-84fd0b79b881","Type":"ContainerStarted","Data":"ed7d6a84d1d1d8842fd473ffe32da59b0e7ee062dc03d932868a95e01a9aac4e"}
Nov 01 05:58:20.316312 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:20.316291    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq" event={"ID":"a82ed2f5-f019-4457-a3f9-b99da85c104f","Type":"ContainerStarted","Data":"a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77"}
Nov 01 05:58:20.316402 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:20.316328    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq" event={"ID":"a82ed2f5-f019-4457-a3f9-b99da85c104f","Type":"ContainerStarted","Data":"dc45f9d23cb32b99391e4d6b794d737fd83bd722e2829340199c8abfc2a8c515"}
Nov 01 05:58:20.316461 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:20.316433    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:20.430159 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:20.430110    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q" podStartSLOduration=2.430094935 podStartE2EDuration="2.430094935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:20.429148077 +0000 UTC m=+737.307121489" watchObservedRunningTime="2026-11-01 05:58:20.430094935 +0000 UTC m=+737.308068341"
Nov 01 05:58:20.552538 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:20.552474    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq" podStartSLOduration=2.552452836 podStartE2EDuration="2.552452836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:20.550104202 +0000 UTC m=+737.428077612" watchObservedRunningTime="2026-11-01 05:58:20.552452836 +0000 UTC m=+737.430426247"
Nov 01 05:58:23.630727 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:23.629874    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:38910->10.134.0.39:15021: read: connection reset by peer" start-of-body=
Nov 01 05:58:23.630727 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:23.629939    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:38910->10.134.0.39:15021: read: connection reset by peer"
Nov 01 05:58:24.339408 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:24.339373    5510 generic.go:356] "Generic (PLEG): container finished" podID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerID="be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f" exitCode=0
Nov 01 05:58:24.339655 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:24.339443    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" event={"ID":"8efa1d66-775f-4ba4-ae2e-658da02e7ad0","Type":"ContainerDied","Data":"be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f"}
Nov 01 05:58:24.339807 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:24.339794    5510 scope.go:122] "RemoveContainer" containerID="be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f"
Nov 01 05:58:24.341102 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:24.341077    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:24.341193 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:24.341141    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:24.341193 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:24.341177    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:25.346262 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:25.346223    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" event={"ID":"8efa1d66-775f-4ba4-ae2e-658da02e7ad0","Type":"ContainerStarted","Data":"a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e"}
Nov 01 05:58:26.068187 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:26.068151    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:58:26.068363 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:26.068320    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:58:26.068415 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:26.068370    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:58:27.078234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:27.078194    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:58:27.078869 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:27.078803    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:58:28.069082 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.069046    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:58:28.069410 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.069377    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:58:28.441978 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.441891    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"]
Nov 01 05:58:28.447214 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.447193    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.501759 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.501728    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"simulator-w0-9c8fc5-kserve-self-signed-certs\""
Nov 01 05:58:28.551479 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.551446    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"default-dockercfg-rjmhh\""
Nov 01 05:58:28.551479 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.551457    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"kube-root-ca.crt\""
Nov 01 05:58:28.551689 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.551514    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"openshift-service-ca.crt\""
Nov 01 05:58:28.552946 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.552928    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"]
Nov 01 05:58:28.614371 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.614340    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-model-cache\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.614569 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.614392    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-dshm\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.614569 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.614430    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-home\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.614664 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.614569    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tls-certs\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.614747 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.614727    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tmp-dir\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.614786 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.614763    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9skg\" (UniqueName: \"kubernetes.io/projected/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-kube-api-access-p9skg\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.715446 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715414    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tmp-dir\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.715630 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715457    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9skg\" (UniqueName: \"kubernetes.io/projected/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-kube-api-access-p9skg\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.715630 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715484    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-model-cache\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.715630 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715506    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-dshm\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.715630 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715526    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-home\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.715630 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715550    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tls-certs\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.715918 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715900    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tmp-dir\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.716004 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715979    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-home\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.716143 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.715977    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-model-cache\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.718228 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.718202    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-dshm\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.718426 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.718412    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tls-certs\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:28.796234 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:28.796195    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9skg\" (UniqueName: \"kubernetes.io/projected/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-kube-api-access-p9skg\") pod \"simulator-w0-9c8fc5-kserve-86f756d849-brfz6\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") " pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:29.067075 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:29.066985    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:29.068232 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:29.068207    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:58:29.068296 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:29.068265    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:58:29.463642 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:29.463582    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"]
Nov 01 05:58:29.466013 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:58:29.465986    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cc62f94_c7cd_4b71_aaa4_ce3b32589ba9.slice/crio-d7e450006d493fbc73ea751b546b8b0f5e927ed6f99e5e8d1854ed33d6c02dfe WatchSource:0}: Error finding container d7e450006d493fbc73ea751b546b8b0f5e927ed6f99e5e8d1854ed33d6c02dfe: Status 404 returned error can't find the container with id d7e450006d493fbc73ea751b546b8b0f5e927ed6f99e5e8d1854ed33d6c02dfe
Nov 01 05:58:30.068990 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:30.068951    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:58:30.069261 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:30.069231    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:58:30.371398 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:30.371309    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6" event={"ID":"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9","Type":"ContainerStarted","Data":"58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33"}
Nov 01 05:58:30.371398 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:30.371363    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6" event={"ID":"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9","Type":"ContainerStarted","Data":"d7e450006d493fbc73ea751b546b8b0f5e927ed6f99e5e8d1854ed33d6c02dfe"}
Nov 01 05:58:30.489060 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:30.488992    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6" podStartSLOduration=2.4889711070000002 podStartE2EDuration="2.488971107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:30.487670871 +0000 UTC m=+747.365644294" watchObservedRunningTime="2026-11-01 05:58:30.488971107 +0000 UTC m=+747.366944519"
Nov 01 05:58:31.067463 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:31.067433    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:31.073536 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:31.073504    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:58:31.075053 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:31.075030    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:58:31.076280 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:31.076263    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 05:58:31.325496 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:31.325409    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 05:58:31.327807 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:31.327781    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 05:58:34.912890 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:34.912860    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:58:38.888722 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.888690    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"]
Nov 01 05:58:38.893922 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.893902    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:38.905684 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.905659    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:38.905818 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.905799    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:38.905885 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.905845    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/a96273ce-152f-4248-88bc-02d9dc6487ba-kube-api-access-wlbqn\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:38.947681 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.947659    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 05:58:38.997202 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.997170    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 05:58:38.997202 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.997198    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 05:58:38.997365 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.997300    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 05:58:38.999127 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:38.999111    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"]
Nov 01 05:58:39.007370 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.007347    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:39.007476 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.007372    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/a96273ce-152f-4248-88bc-02d9dc6487ba-kube-api-access-wlbqn\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:39.007476 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.007415    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:39.008049 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.008032    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:39.009807 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.009788    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:39.067510 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.067480    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:39.067900 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.067875    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/a96273ce-152f-4248-88bc-02d9dc6487ba-kube-api-access-wlbqn\") pod \"maas-api-revoke-sub-9b59859b-d54nb\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:39.079901 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.079882    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:39.207761 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.207739    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:39.426432 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.426399    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 05:58:39.593439 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:39.593371    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"]
Nov 01 05:58:39.620139 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:58:39.620113    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96273ce_152f_4248_88bc_02d9dc6487ba.slice/crio-2967df3cdbd5fd7eaed4d52371ac02cbd2efff812861a5f799b188eaf726d450 WatchSource:0}: Error finding container 2967df3cdbd5fd7eaed4d52371ac02cbd2efff812861a5f799b188eaf726d450: Status 404 returned error can't find the container with id 2967df3cdbd5fd7eaed4d52371ac02cbd2efff812861a5f799b188eaf726d450
Nov 01 05:58:40.421345 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:40.421305    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb" event={"ID":"a96273ce-152f-4248-88bc-02d9dc6487ba","Type":"ContainerStarted","Data":"2967df3cdbd5fd7eaed4d52371ac02cbd2efff812861a5f799b188eaf726d450"}
Nov 01 05:58:44.444442 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:44.444407    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb" event={"ID":"a96273ce-152f-4248-88bc-02d9dc6487ba","Type":"ContainerStarted","Data":"56dd02096f738501c738e0ddda5b426f8c53a3ad2d883aac777bbb80c24f9459"}
Nov 01 05:58:44.559560 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:44.559509    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb" podStartSLOduration=2.274507507 podStartE2EDuration="6.559493568s" totalImagesPullingTime="4.284986061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:44.558848726 +0000 UTC m=+761.436822132" watchObservedRunningTime="2026-11-01 05:58:44.559493568 +0000 UTC m=+761.437466978"
Nov 01 05:58:45.450349 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:45.450314    5510 generic.go:356] "Generic (PLEG): container finished" podID="a96273ce-152f-4248-88bc-02d9dc6487ba" containerID="56dd02096f738501c738e0ddda5b426f8c53a3ad2d883aac777bbb80c24f9459" exitCode=0
Nov 01 05:58:45.450751 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:45.450404    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb" event={"ID":"a96273ce-152f-4248-88bc-02d9dc6487ba","Type":"ContainerDied","Data":"56dd02096f738501c738e0ddda5b426f8c53a3ad2d883aac777bbb80c24f9459"}
Nov 01 05:58:46.631580 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.631558    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:46.707378 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.707328    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-xks-ca\") pod \"a96273ce-152f-4248-88bc-02d9dc6487ba\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") "
Nov 01 05:58:46.707378 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.707371    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-openshift-ca\") pod \"a96273ce-152f-4248-88bc-02d9dc6487ba\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") "
Nov 01 05:58:46.707575 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.707400    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96273ce-152f-4248-88bc-02d9dc6487ba-kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/a96273ce-152f-4248-88bc-02d9dc6487ba-kube-api-access-wlbqn\") pod \"a96273ce-152f-4248-88bc-02d9dc6487ba\" (UID: \"a96273ce-152f-4248-88bc-02d9dc6487ba\") "
Nov 01 05:58:46.707743 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.707718    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-openshift-ca" pod "a96273ce-152f-4248-88bc-02d9dc6487ba" (UID: "a96273ce-152f-4248-88bc-02d9dc6487ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:58:46.709863 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.709818    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-xks-ca" pod "a96273ce-152f-4248-88bc-02d9dc6487ba" (UID: "a96273ce-152f-4248-88bc-02d9dc6487ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:58:46.709863 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.709817    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96273ce-152f-4248-88bc-02d9dc6487ba-kube-api-access-wlbqn" pod "a96273ce-152f-4248-88bc-02d9dc6487ba" (UID: "a96273ce-152f-4248-88bc-02d9dc6487ba"). InnerVolumeSpecName "kube-api-access-wlbqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:58:46.808231 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.808198    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:58:46.808231 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.808233    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96273ce-152f-4248-88bc-02d9dc6487ba-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:58:46.808467 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:46.808250    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/a96273ce-152f-4248-88bc-02d9dc6487ba-kube-api-access-wlbqn\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:58:47.460524 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:47.460500    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"
Nov 01 05:58:47.460689 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:47.460542    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb" event={"ID":"a96273ce-152f-4248-88bc-02d9dc6487ba","Type":"ContainerDied","Data":"2967df3cdbd5fd7eaed4d52371ac02cbd2efff812861a5f799b188eaf726d450"}
Nov 01 05:58:47.460689 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:47.460601    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2967df3cdbd5fd7eaed4d52371ac02cbd2efff812861a5f799b188eaf726d450"
Nov 01 05:58:47.545391 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:58:47.545354    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:58:47.690728 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:47.690694    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:58:49.266985 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:49.266941    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:58:49.267369 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:49.266994    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:58:51.484303 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:51.484271    5510 generic.go:356] "Generic (PLEG): container finished" podID="75de03a3-8add-411c-a293-9487c9b218ad" containerID="a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac" exitCode=0
Nov 01 05:58:51.484740 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:51.484343    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" event={"ID":"75de03a3-8add-411c-a293-9487c9b218ad","Type":"ContainerDied","Data":"a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac"}
Nov 01 05:58:51.484740 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:51.484708    5510 scope.go:122] "RemoveContainer" containerID="a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac"
Nov 01 05:58:51.485890 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:51.485858    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:51.485980 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:51.485927    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:51.485980 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:51.485963    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:52.499176 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:52.499143    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" event={"ID":"75de03a3-8add-411c-a293-9487c9b218ad","Type":"ContainerStarted","Data":"5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a"}
Nov 01 05:58:53.088020 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:53.087989    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:53.088653 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:53.088628    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:58:53.088891 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:53.088864    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:58:54.013199 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.013163    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"]
Nov 01 05:58:54.013620 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.013607    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a96273ce-152f-4248-88bc-02d9dc6487ba" containerName="revoke-keys"
Nov 01 05:58:54.013674 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.013621    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96273ce-152f-4248-88bc-02d9dc6487ba" containerName="revoke-keys"
Nov 01 05:58:54.013739 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.013729    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96273ce-152f-4248-88bc-02d9dc6487ba" containerName="revoke-keys"
Nov 01 05:58:54.020014 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.019994    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.089693 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.089661    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:58:54.089988 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.089958    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:58:54.107665 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.107644    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.107791 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.107671    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fa3477-782d-4d47-b594-cfa53dc6af05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.107791 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.107694    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.107791 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.107717    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.107950 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.107819    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.107950 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.107923    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.108058 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.107976    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.108115 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.108050    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gnk\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-kube-api-access-c4gnk\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.108186 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.108167    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.118743 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.118719    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"]
Nov 01 05:58:54.209382 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209345    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.209526 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209393    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.209526 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209417    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fa3477-782d-4d47-b594-cfa53dc6af05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.209637 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209585    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.209637 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209628    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.209755 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209664    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.209755 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209707    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.209916 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209764    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.209916 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.209808    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gnk\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-kube-api-access-c4gnk\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.210033 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.210013    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.210748 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.210059    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.210992 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.210747    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fa3477-782d-4d47-b594-cfa53dc6af05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.211114 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.211075    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.211262 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.211234    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.212510 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.212485    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.215744 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.213090    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.296461 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.296381    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.296675 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.296521    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gnk\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-kube-api-access-c4gnk\") pod \"maas-default-gateway-openshift-default-855f9574cf-fj6vk\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.333642 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.333608    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:54.784871 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.781988    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"]
Nov 01 05:58:54.795077 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:58:54.795044    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fa3477_782d_4d47_b594_cfa53dc6af05.slice/crio-9f4c3f806268ddf2012209da63c05cb6579f59b5a4216c655886023ac2c9c72e WatchSource:0}: Error finding container 9f4c3f806268ddf2012209da63c05cb6579f59b5a4216c655886023ac2c9c72e: Status 404 returned error can't find the container with id 9f4c3f806268ddf2012209da63c05cb6579f59b5a4216c655886023ac2c9c72e
Nov 01 05:58:54.798045 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.797692    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:54.798045 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.797762    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:54.798045 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:54.797818    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:58:55.087902 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:55.087206    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:58:55.087902 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:55.087264    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:58:55.522780 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:55.522741    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" event={"ID":"69fa3477-782d-4d47-b594-cfa53dc6af05","Type":"ContainerStarted","Data":"0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d"}
Nov 01 05:58:55.523017 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:55.522794    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" event={"ID":"69fa3477-782d-4d47-b594-cfa53dc6af05","Type":"ContainerStarted","Data":"9f4c3f806268ddf2012209da63c05cb6579f59b5a4216c655886023ac2c9c72e"}
Nov 01 05:58:55.651536 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:55.651483    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podStartSLOduration=2.651467989 podStartE2EDuration="2.651467989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:55.649093282 +0000 UTC m=+772.527066693" watchObservedRunningTime="2026-11-01 05:58:55.651467989 +0000 UTC m=+772.529441402"
Nov 01 05:58:56.090853 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:56.086228    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:58:56.090853 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:56.086292    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:58:56.333850 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:56.333797    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:58:56.335491 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:56.335465    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fj6vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 05:58:56.335607 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:56.335513    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 05:58:57.085240 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:57.085200    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:58:57.085535 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:57.085507    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:58:57.334592 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:57.334563    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fj6vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 05:58:57.335022 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:57.334624    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 05:58:58.116008 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:58.115972    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:58.117305 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:58.117281    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:58.118020 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:58.117998    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 05:58:58.334396 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:58.334362    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fj6vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 05:58:58.334575 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:58.334425    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 05:58:59.334712 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:59.334674    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fj6vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 05:58:59.335144 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:58:59.334734    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 05:59:00.334161 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:00.334127    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fj6vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 05:59:00.334333 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:00.334175    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 05:59:01.119631 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:01.119589    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:59:01.334938 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:01.334899    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fj6vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 05:59:01.335138 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:01.334948    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 05:59:02.334971 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.334939    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fj6vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 05:59:02.335485 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.334996    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 05:59:02.352044 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.352008    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"]
Nov 01 05:59:02.358064 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.358042    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.418698 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.418669    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"]
Nov 01 05:59:02.432499 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.432471    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-home\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.432676 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.432504    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-model-cache\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.432676 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.432582    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmr5\" (UniqueName: \"kubernetes.io/projected/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-kube-api-access-mqmr5\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.432676 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.432615    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-dshm\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.432804 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.432704    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tls-certs\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.432804 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.432736    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tmp-dir\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.441255 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.441232    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9c8fc5\"/\"premium-w0-9c8fc5-kserve-self-signed-certs\""
Nov 01 05:59:02.533700 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.533668    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tls-certs\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.533952 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.533709    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tmp-dir\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.533952 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.533759    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-home\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.533952 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.533880    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-model-cache\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.534142 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.533958    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmr5\" (UniqueName: \"kubernetes.io/projected/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-kube-api-access-mqmr5\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.534142 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.534011    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-dshm\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.534251 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.534170    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-home\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.534251 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.534228    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tmp-dir\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.534466 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.534442    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-model-cache\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.536382 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.536353    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-dshm\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.536493 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.536399    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tls-certs\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.597312 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.597241    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmr5\" (UniqueName: \"kubernetes.io/projected/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-kube-api-access-mqmr5\") pod \"premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") " pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:02.670519 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:02.670473    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:03.066850 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:03.064943    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"]
Nov 01 05:59:03.068336 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:59:03.068307    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccd4246d_28af_4510_a7cd_d02e63d6fb8b.slice/crio-a9ef0f23671a45594732a59be009f1a79b88864f9f01ba6d973e6f7975af6541 WatchSource:0}: Error finding container a9ef0f23671a45594732a59be009f1a79b88864f9f01ba6d973e6f7975af6541: Status 404 returned error can't find the container with id a9ef0f23671a45594732a59be009f1a79b88864f9f01ba6d973e6f7975af6541
Nov 01 05:59:03.575482 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:03.575441    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7" event={"ID":"ccd4246d-28af-4510-a7cd-d02e63d6fb8b","Type":"ContainerStarted","Data":"21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2"}
Nov 01 05:59:03.575482 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:03.575484    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7" event={"ID":"ccd4246d-28af-4510-a7cd-d02e63d6fb8b","Type":"ContainerStarted","Data":"a9ef0f23671a45594732a59be009f1a79b88864f9f01ba6d973e6f7975af6541"}
Nov 01 05:59:03.586856 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:03.586812    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:59:03.587154 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:03.587135    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:59:03.587997 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:03.587979    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 05:59:03.670745 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:03.670717    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:03.692360 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:03.692315    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7" podStartSLOduration=1.692301217 podStartE2EDuration="1.692301217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:03.690399284 +0000 UTC m=+780.568372694" watchObservedRunningTime="2026-11-01 05:59:03.692301217 +0000 UTC m=+780.570274635"
Nov 01 05:59:05.916906 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:05.916873    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"]
Nov 01 05:59:05.920803 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:05.920788    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:06.025651 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.024966    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0d2807-dockercfg-rpr7q\""
Nov 01 05:59:06.025651 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.025254    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0d2807\""
Nov 01 05:59:06.028247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.028225    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"]
Nov 01 05:59:06.073125 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.073083    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw\" (UID: \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:06.073268 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.073225    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n428x\" (UniqueName: \"kubernetes.io/projected/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-kube-api-access-n428x\") pod \"payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw\" (UID: \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:06.173988 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.173913    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw\" (UID: \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:06.174161 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.174004    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n428x\" (UniqueName: \"kubernetes.io/projected/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-kube-api-access-n428x\") pod \"payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw\" (UID: \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:06.174683 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.174657    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw\" (UID: \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:06.235011 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.234986    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n428x\" (UniqueName: \"kubernetes.io/projected/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-kube-api-access-n428x\") pod \"payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw\" (UID: \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:06.534246 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.534216    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:06.911154 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:06.911130    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"]
Nov 01 05:59:06.913270 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:59:06.913239    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22f3138_7e24_4e67_a572_10e9e1d2a8e9.slice/crio-3404d247fccb3d17332fec7a3a447ec67403173e139376b437ddbe94fc030084 WatchSource:0}: Error finding container 3404d247fccb3d17332fec7a3a447ec67403173e139376b437ddbe94fc030084: Status 404 returned error can't find the container with id 3404d247fccb3d17332fec7a3a447ec67403173e139376b437ddbe94fc030084
Nov 01 05:59:07.597981 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:07.597952    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw" event={"ID":"f22f3138-7e24-4e67-a572-10e9e1d2a8e9","Type":"ContainerStarted","Data":"848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4"}
Nov 01 05:59:07.598383 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:07.598002    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw" event={"ID":"f22f3138-7e24-4e67-a572-10e9e1d2a8e9","Type":"ContainerStarted","Data":"3404d247fccb3d17332fec7a3a447ec67403173e139376b437ddbe94fc030084"}
Nov 01 05:59:07.598383 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:07.598023    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:07.714261 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:07.714210    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw" podStartSLOduration=2.714195485 podStartE2EDuration="2.714195485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:07.712020408 +0000 UTC m=+784.589993820" watchObservedRunningTime="2026-11-01 05:59:07.714195485 +0000 UTC m=+784.592168895"
Nov 01 05:59:09.023994 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.023962    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"]
Nov 01 05:59:09.027655 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.027640    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104420 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104392    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104551 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104438    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104551 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104529    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104630 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104571    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104630 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104607    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104630 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104622    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104759 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104743    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104794 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104772    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7lq\" (UniqueName: \"kubernetes.io/projected/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-kube-api-access-kn7lq\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.104838 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.104793    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.131751 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.131721    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"]
Nov 01 05:59:09.205655 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.205614    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.205816 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.205660    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7lq\" (UniqueName: \"kubernetes.io/projected/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-kube-api-access-kn7lq\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.205816 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.205680    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.205816 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.205714    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.205816 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.205745    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.205816 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.205782    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.206072 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.205941    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.206072 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.206025    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.206297 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.206065    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.206297 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.206073    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.206297 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.206129    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.206297 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.206167    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.206510 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.206486    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.206985 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.206962    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.208272 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.208247    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.208451 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.208433    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.266422 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.266393    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.266570 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.266426    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7lq\" (UniqueName: \"kubernetes.io/projected/ceddee3f-6781-46eb-ab9b-9c9eb267d18b-kube-api-access-kn7lq\") pod \"maas-default-gateway-openshift-default-855f9574cf-6h48b\" (UID: \"ceddee3f-6781-46eb-ab9b-9c9eb267d18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.340327 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.340240    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:09.765110 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.765084    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"]
Nov 01 05:59:09.767252 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:59:09.767226    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceddee3f_6781_46eb_ab9b_9c9eb267d18b.slice/crio-428a87f033d72413143ce1ecc5dc64d50591ae7f864b6c9c683683eacb933643 WatchSource:0}: Error finding container 428a87f033d72413143ce1ecc5dc64d50591ae7f864b6c9c683683eacb933643: Status 404 returned error can't find the container with id 428a87f033d72413143ce1ecc5dc64d50591ae7f864b6c9c683683eacb933643
Nov 01 05:59:09.769632 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.769590    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:59:09.769725 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.769683    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:59:09.769761 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:09.769724    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 05:59:10.619220 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:10.619178    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" event={"ID":"ceddee3f-6781-46eb-ab9b-9c9eb267d18b","Type":"ContainerStarted","Data":"b00dc34e6b9315496381e5dee7966f2b51c7fe1c34df0489335675cc0b7fc01e"}
Nov 01 05:59:10.619752 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:10.619253    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" event={"ID":"ceddee3f-6781-46eb-ab9b-9c9eb267d18b","Type":"ContainerStarted","Data":"428a87f033d72413143ce1ecc5dc64d50591ae7f864b6c9c683683eacb933643"}
Nov 01 05:59:10.842967 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:10.842917    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podStartSLOduration=2.842901593 podStartE2EDuration="2.842901593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:10.793011913 +0000 UTC m=+787.670985354" watchObservedRunningTime="2026-11-01 05:59:10.842901593 +0000 UTC m=+787.720875003"
Nov 01 05:59:11.341247 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:11.341218    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:11.342747 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:11.342718    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:59:11.342917 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:11.342764    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:59:12.341417 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:12.341387    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:59:12.341794 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:12.341444    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:59:12.681907 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:12.679900    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:12.702745 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:12.702717    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:13.341541 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:13.341504    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:59:13.341958 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:13.341557    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:59:13.644708 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:13.644627    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 05:59:14.340953 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:14.340916    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:59:14.341130 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:14.340976    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:59:15.341056 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:15.341021    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:59:15.341443 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:15.341080    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:59:16.341175 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:16.341137    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:59:16.341602 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:16.341216    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:59:16.689335 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:16.689255    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:59:17.341571 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:17.341539    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:59:17.341977 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:17.341590    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:59:17.545537 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:59:17.545490    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:59:18.411136 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:18.411096    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:18.411629 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:18.411476    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:18.412753 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:18.412727    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b"
Nov 01 05:59:18.606235 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:18.606199    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 05:59:31.015251 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:31.011127    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:59:41.153990 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.153953    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"]
Nov 01 05:59:41.158863 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.158820    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.231196 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.231156    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"openshift-service-ca.crt\""
Nov 01 05:59:41.238328 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.238303    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"]
Nov 01 05:59:41.241916 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.241889    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5db556-563e-43f5-ab8a-30d2b3df2254-tls-certs\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.242108 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.242088    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-home\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.242193 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.242177    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-dshm\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.242271 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.242252    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-tmp-dir\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.242328 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.242291    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/bb5db556-563e-43f5-ab8a-30d2b3df2254-kube-api-access-7jtj9\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.242328 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.242319    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-model-cache\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.263325 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.263299    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"kube-root-ca.crt\""
Nov 01 05:59:41.263422 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.263407    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"default-dockercfg-262xs\""
Nov 01 05:59:41.263422 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.263416    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"premium-w4-f73e89-kserve-self-signed-certs\""
Nov 01 05:59:41.343579 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.343546    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5db556-563e-43f5-ab8a-30d2b3df2254-tls-certs\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.343787 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.343633    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-home\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.343787 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.343667    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-dshm\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.343787 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.343689    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-tmp-dir\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.343787 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.343710    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/bb5db556-563e-43f5-ab8a-30d2b3df2254-kube-api-access-7jtj9\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.343787 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.343730    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-model-cache\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.344163 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.344128    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-model-cache\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.344224 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.344170    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-tmp-dir\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.344224 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.344210    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-home\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.345995 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.345974    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-dshm\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.346464 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.346443    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5db556-563e-43f5-ab8a-30d2b3df2254-tls-certs\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.404489 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.404430    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/bb5db556-563e-43f5-ab8a-30d2b3df2254-kube-api-access-7jtj9\") pod \"premium-w4-f73e89-kserve-54578f8898-vzxkv\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") " pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.493302 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.493272    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:41.900392 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:41.900368    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"]
Nov 01 05:59:41.903124 ip-10-0-138-190 kubenswrapper[5510]: W1101 05:59:41.903097    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb5db556_563e_43f5_ab8a_30d2b3df2254.slice/crio-1de5cacc1845d6aec1191937cef6a57e39cbc0e860d1fed170db32c897a45aa4 WatchSource:0}: Error finding container 1de5cacc1845d6aec1191937cef6a57e39cbc0e860d1fed170db32c897a45aa4: Status 404 returned error can't find the container with id 1de5cacc1845d6aec1191937cef6a57e39cbc0e860d1fed170db32c897a45aa4
Nov 01 05:59:42.802670 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:42.802634    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv" event={"ID":"bb5db556-563e-43f5-ab8a-30d2b3df2254","Type":"ContainerStarted","Data":"5c1f51102210ba6ad04bbbb6d52fad211237a6e641a52afc1ae26db748a61564"}
Nov 01 05:59:42.803093 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:42.802678    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv" event={"ID":"bb5db556-563e-43f5-ab8a-30d2b3df2254","Type":"ContainerStarted","Data":"1de5cacc1845d6aec1191937cef6a57e39cbc0e860d1fed170db32c897a45aa4"}
Nov 01 05:59:43.006795 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:43.006745    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv" podStartSLOduration=2.006731106 podStartE2EDuration="2.006731106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:43.005378942 +0000 UTC m=+819.883352353" watchObservedRunningTime="2026-11-01 05:59:43.006731106 +0000 UTC m=+819.884704515"
Nov 01 05:59:43.493403 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:43.493368    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:46.728765 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:46.728722    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 05:59:47.544820 ip-10-0-138-190 kubenswrapper[5510]: E1101 05:59:47.544780    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 05:59:51.493869 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:51.493808    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:51.506592 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:51.506570    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 05:59:51.863015 ip-10-0-138-190 kubenswrapper[5510]: I1101 05:59:51.862934    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 06:00:00.168737 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.168698    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"]
Nov 01 06:00:00.172808 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.172789    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"]
Nov 01 06:00:00.172954 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.172920    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:00.176475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.176458    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"]
Nov 01 06:00:00.176609 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.176591    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:00.179739 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.179724    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:00.225006 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.224981    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:00:00.225163 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.225059    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:00:00.225163 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.225106    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:00:00.234041 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.234019    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f9f806-fc7c-4029-9e80-1058f4339e01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g\" (UID: \"80f9f806-fc7c-4029-9e80-1058f4339e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:00.234154 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.234087    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2plvm\" (UniqueName: \"kubernetes.io/projected/1556ac3f-d13e-47ed-b971-308e9cf90ce8-kube-api-access-2plvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs\" (UID: \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:00.234216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.234199    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891880-g4gq2\" (UID: \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:00.234261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.234227    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkgg\" (UniqueName: \"kubernetes.io/projected/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-kube-api-access-wlkgg\") pod \"maas-api-key-cleanup-29891880-g4gq2\" (UID: \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:00.234295 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.234267    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1556ac3f-d13e-47ed-b971-308e9cf90ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs\" (UID: \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:00.234338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.234325    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v7bx\" (UniqueName: \"kubernetes.io/projected/80f9f806-fc7c-4029-9e80-1058f4339e01-kube-api-access-6v7bx\") pod \"maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g\" (UID: \"80f9f806-fc7c-4029-9e80-1058f4339e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:00.276530 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.276498    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"]
Nov 01 06:00:00.279077 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.279046    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"]
Nov 01 06:00:00.335111 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.335079    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f9f806-fc7c-4029-9e80-1058f4339e01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g\" (UID: \"80f9f806-fc7c-4029-9e80-1058f4339e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:00.335317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.335174    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2plvm\" (UniqueName: \"kubernetes.io/projected/1556ac3f-d13e-47ed-b971-308e9cf90ce8-kube-api-access-2plvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs\" (UID: \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:00.335317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.335236    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891880-g4gq2\" (UID: \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:00.335317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.335266    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkgg\" (UniqueName: \"kubernetes.io/projected/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-kube-api-access-wlkgg\") pod \"maas-api-key-cleanup-29891880-g4gq2\" (UID: \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:00.335317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.335316    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1556ac3f-d13e-47ed-b971-308e9cf90ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs\" (UID: \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:00.335543 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.335347    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v7bx\" (UniqueName: \"kubernetes.io/projected/80f9f806-fc7c-4029-9e80-1058f4339e01-kube-api-access-6v7bx\") pod \"maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g\" (UID: \"80f9f806-fc7c-4029-9e80-1058f4339e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:00.335996 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.335974    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f9f806-fc7c-4029-9e80-1058f4339e01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g\" (UID: \"80f9f806-fc7c-4029-9e80-1058f4339e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:00.336109 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.335996    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1556ac3f-d13e-47ed-b971-308e9cf90ce8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs\" (UID: \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:00.336109 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.336016    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891880-g4gq2\" (UID: \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:00.342197 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.342172    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"]
Nov 01 06:00:00.395866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.395822    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkgg\" (UniqueName: \"kubernetes.io/projected/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-kube-api-access-wlkgg\") pod \"maas-api-key-cleanup-29891880-g4gq2\" (UID: \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:00.396005 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.395948    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v7bx\" (UniqueName: \"kubernetes.io/projected/80f9f806-fc7c-4029-9e80-1058f4339e01-kube-api-access-6v7bx\") pod \"maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g\" (UID: \"80f9f806-fc7c-4029-9e80-1058f4339e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:00.396101 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.396081    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2plvm\" (UniqueName: \"kubernetes.io/projected/1556ac3f-d13e-47ed-b971-308e9cf90ce8-kube-api-access-2plvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs\" (UID: \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:00.486604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.486573    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:00.495347 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.495323    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:00.504640 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.504605    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:00.868701 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.868626    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"]
Nov 01 06:00:00.886334 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:00:00.886306    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1556ac3f_d13e_47ed_b971_308e9cf90ce8.slice/crio-3b59489e1e12f44599f25bb1019516eb05108a1d82d934473253e0cd01a2bc9b WatchSource:0}: Error finding container 3b59489e1e12f44599f25bb1019516eb05108a1d82d934473253e0cd01a2bc9b: Status 404 returned error can't find the container with id 3b59489e1e12f44599f25bb1019516eb05108a1d82d934473253e0cd01a2bc9b
Nov 01 06:00:00.900212 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.900187    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" event={"ID":"1556ac3f-d13e-47ed-b971-308e9cf90ce8","Type":"ContainerStarted","Data":"3b59489e1e12f44599f25bb1019516eb05108a1d82d934473253e0cd01a2bc9b"}
Nov 01 06:00:00.919244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.919204    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"]
Nov 01 06:00:00.920986 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:00.920964    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"]
Nov 01 06:00:00.921370 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:00:00.921348    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ebac6e_9cf5_4150_91ba_7b1b0b9f1870.slice/crio-93e3db51f01897b2bffeb08c9643f4156417e85bc1ca0c52852a43792e697633 WatchSource:0}: Error finding container 93e3db51f01897b2bffeb08c9643f4156417e85bc1ca0c52852a43792e697633: Status 404 returned error can't find the container with id 93e3db51f01897b2bffeb08c9643f4156417e85bc1ca0c52852a43792e697633
Nov 01 06:00:00.922153 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:00:00.922125    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f9f806_fc7c_4029_9e80_1058f4339e01.slice/crio-d211701acab8edbd3b33eb7d75b5ef20c5cf6357b8f95b76ccda8203a5eb6d8e WatchSource:0}: Error finding container d211701acab8edbd3b33eb7d75b5ef20c5cf6357b8f95b76ccda8203a5eb6d8e: Status 404 returned error can't find the container with id d211701acab8edbd3b33eb7d75b5ef20c5cf6357b8f95b76ccda8203a5eb6d8e
Nov 01 06:00:01.906785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.906758    5510 generic.go:356] "Generic (PLEG): container finished" podID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerID="b9ffa2facd3b14ccb308830ca5cd9beecf5ec022059ce49e8a9d4f8bfa6f83d7" exitCode=60
Nov 01 06:00:01.907208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.906844    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" event={"ID":"1556ac3f-d13e-47ed-b971-308e9cf90ce8","Type":"ContainerDied","Data":"b9ffa2facd3b14ccb308830ca5cd9beecf5ec022059ce49e8a9d4f8bfa6f83d7"}
Nov 01 06:00:01.907208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.907111    5510 scope.go:122] "RemoveContainer" containerID="b9ffa2facd3b14ccb308830ca5cd9beecf5ec022059ce49e8a9d4f8bfa6f83d7"
Nov 01 06:00:01.908350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.908329    5510 generic.go:356] "Generic (PLEG): container finished" podID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerID="aabc51cb07a83da38d3fdf03c2c2b0b38e017e465d265220d4001f06272d6d41" exitCode=60
Nov 01 06:00:01.908441 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.908392    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" event={"ID":"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870","Type":"ContainerDied","Data":"aabc51cb07a83da38d3fdf03c2c2b0b38e017e465d265220d4001f06272d6d41"}
Nov 01 06:00:01.908441 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.908436    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" event={"ID":"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870","Type":"ContainerStarted","Data":"93e3db51f01897b2bffeb08c9643f4156417e85bc1ca0c52852a43792e697633"}
Nov 01 06:00:01.908569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.908553    5510 scope.go:122] "RemoveContainer" containerID="aabc51cb07a83da38d3fdf03c2c2b0b38e017e465d265220d4001f06272d6d41"
Nov 01 06:00:01.910173 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.910118    5510 generic.go:356] "Generic (PLEG): container finished" podID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerID="7701310c6c1110d4c6fd2e00bd8c0393cb72f389d0cb1d3c449f0aa1a5e72a00" exitCode=60
Nov 01 06:00:01.910173 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.910154    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" event={"ID":"80f9f806-fc7c-4029-9e80-1058f4339e01","Type":"ContainerDied","Data":"7701310c6c1110d4c6fd2e00bd8c0393cb72f389d0cb1d3c449f0aa1a5e72a00"}
Nov 01 06:00:01.910237 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.910202    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" event={"ID":"80f9f806-fc7c-4029-9e80-1058f4339e01","Type":"ContainerStarted","Data":"d211701acab8edbd3b33eb7d75b5ef20c5cf6357b8f95b76ccda8203a5eb6d8e"}
Nov 01 06:00:01.910413 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:01.910402    5510 scope.go:122] "RemoveContainer" containerID="7701310c6c1110d4c6fd2e00bd8c0393cb72f389d0cb1d3c449f0aa1a5e72a00"
Nov 01 06:00:02.918033 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:02.917997    5510 generic.go:356] "Generic (PLEG): container finished" podID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerID="c1ceb0e212fb8c66cfe03aaddab6afb380d255f0307c72572740c685d14cebd9" exitCode=60
Nov 01 06:00:02.918526 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:02.918079    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" event={"ID":"1556ac3f-d13e-47ed-b971-308e9cf90ce8","Type":"ContainerDied","Data":"c1ceb0e212fb8c66cfe03aaddab6afb380d255f0307c72572740c685d14cebd9"}
Nov 01 06:00:02.918526 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:02.918129    5510 scope.go:122] "RemoveContainer" containerID="b9ffa2facd3b14ccb308830ca5cd9beecf5ec022059ce49e8a9d4f8bfa6f83d7"
Nov 01 06:00:02.918526 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:02.918403    5510 scope.go:122] "RemoveContainer" containerID="c1ceb0e212fb8c66cfe03aaddab6afb380d255f0307c72572740c685d14cebd9"
Nov 01 06:00:02.918728 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:02.918705    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs_odh-ai-gateway-infra(1556ac3f-d13e-47ed-b971-308e9cf90ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8"
Nov 01 06:00:02.920026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:02.920001    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" event={"ID":"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870","Type":"ContainerStarted","Data":"20026527bb233e0fc07c428e51b5012d8cf726ada9207bec22d8696f61107044"}
Nov 01 06:00:02.921728 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:02.921706    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" event={"ID":"80f9f806-fc7c-4029-9e80-1058f4339e01","Type":"ContainerStarted","Data":"4f95a8ab699dcf913b20085f9cc6c7a4c2b68ff6507355c39cdb181f4ccd4063"}
Nov 01 06:00:03.149119 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.149065    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" podStartSLOduration=3.149051687 podStartE2EDuration="3.149051687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:03.147995482 +0000 UTC m=+840.025968894" watchObservedRunningTime="2026-11-01 06:00:03.149051687 +0000 UTC m=+840.027025093"
Nov 01 06:00:03.264697 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.264649    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" podStartSLOduration=3.264635735 podStartE2EDuration="3.264635735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:03.263054491 +0000 UTC m=+840.141027904" watchObservedRunningTime="2026-11-01 06:00:03.264635735 +0000 UTC m=+840.142609192"
Nov 01 06:00:03.938852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.937710    5510 scope.go:122] "RemoveContainer" containerID="c1ceb0e212fb8c66cfe03aaddab6afb380d255f0307c72572740c685d14cebd9"
Nov 01 06:00:03.938852 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:03.938137    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs_odh-ai-gateway-infra(1556ac3f-d13e-47ed-b971-308e9cf90ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8"
Nov 01 06:00:03.945849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.941341    5510 generic.go:356] "Generic (PLEG): container finished" podID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerID="20026527bb233e0fc07c428e51b5012d8cf726ada9207bec22d8696f61107044" exitCode=60
Nov 01 06:00:03.945849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.941436    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" event={"ID":"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870","Type":"ContainerDied","Data":"20026527bb233e0fc07c428e51b5012d8cf726ada9207bec22d8696f61107044"}
Nov 01 06:00:03.945849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.941489    5510 scope.go:122] "RemoveContainer" containerID="aabc51cb07a83da38d3fdf03c2c2b0b38e017e465d265220d4001f06272d6d41"
Nov 01 06:00:03.945849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.941918    5510 scope.go:122] "RemoveContainer" containerID="20026527bb233e0fc07c428e51b5012d8cf726ada9207bec22d8696f61107044"
Nov 01 06:00:03.948594 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:03.946789    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891880-g4gq2_odh-ai-gateway-infra(c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870"
Nov 01 06:00:03.948594 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.948104    5510 generic.go:356] "Generic (PLEG): container finished" podID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerID="4f95a8ab699dcf913b20085f9cc6c7a4c2b68ff6507355c39cdb181f4ccd4063" exitCode=60
Nov 01 06:00:03.948594 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.948329    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" event={"ID":"80f9f806-fc7c-4029-9e80-1058f4339e01","Type":"ContainerDied","Data":"4f95a8ab699dcf913b20085f9cc6c7a4c2b68ff6507355c39cdb181f4ccd4063"}
Nov 01 06:00:03.948865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.948848    5510 scope.go:122] "RemoveContainer" containerID="4f95a8ab699dcf913b20085f9cc6c7a4c2b68ff6507355c39cdb181f4ccd4063"
Nov 01 06:00:03.949280 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:03.949257    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g_odh-ai-gateway-infra(80f9f806-fc7c-4029-9e80-1058f4339e01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01"
Nov 01 06:00:03.960325 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:03.960303    5510 scope.go:122] "RemoveContainer" containerID="7701310c6c1110d4c6fd2e00bd8c0393cb72f389d0cb1d3c449f0aa1a5e72a00"
Nov 01 06:00:04.955995 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:04.955964    5510 scope.go:122] "RemoveContainer" containerID="20026527bb233e0fc07c428e51b5012d8cf726ada9207bec22d8696f61107044"
Nov 01 06:00:04.956470 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:04.956257    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891880-g4gq2_odh-ai-gateway-infra(c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870"
Nov 01 06:00:04.957449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:04.957433    5510 scope.go:122] "RemoveContainer" containerID="4f95a8ab699dcf913b20085f9cc6c7a4c2b68ff6507355c39cdb181f4ccd4063"
Nov 01 06:00:04.957668 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:04.957651    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g_odh-ai-gateway-infra(80f9f806-fc7c-4029-9e80-1058f4339e01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01"
Nov 01 06:00:11.875776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:11.875740    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:00:12.450164 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.450122    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"]
Nov 01 06:00:12.455101 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.455080    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:12.510047 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.510010    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cda04f\""
Nov 01 06:00:12.512365 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.512335    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"]
Nov 01 06:00:12.600857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.599400    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27p54\" (UniqueName: \"kubernetes.io/projected/11a54968-bcf4-4f36-97a7-86b3c2d15306-kube-api-access-27p54\") pod \"payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k\" (UID: \"11a54968-bcf4-4f36-97a7-86b3c2d15306\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:12.600857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.599526    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a54968-bcf4-4f36-97a7-86b3c2d15306-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k\" (UID: \"11a54968-bcf4-4f36-97a7-86b3c2d15306\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:12.618852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.618044    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"]
Nov 01 06:00:12.626867 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.625915    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:12.655074 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.654985    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"]
Nov 01 06:00:12.679938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.679895    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cda04f-dockercfg-9s7qh\""
Nov 01 06:00:12.702933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.700942    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a54968-bcf4-4f36-97a7-86b3c2d15306-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k\" (UID: \"11a54968-bcf4-4f36-97a7-86b3c2d15306\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:12.702933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.701068    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27p54\" (UniqueName: \"kubernetes.io/projected/11a54968-bcf4-4f36-97a7-86b3c2d15306-kube-api-access-27p54\") pod \"payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k\" (UID: \"11a54968-bcf4-4f36-97a7-86b3c2d15306\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:12.702933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.701925    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a54968-bcf4-4f36-97a7-86b3c2d15306-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k\" (UID: \"11a54968-bcf4-4f36-97a7-86b3c2d15306\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:12.762062 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.762034    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27p54\" (UniqueName: \"kubernetes.io/projected/11a54968-bcf4-4f36-97a7-86b3c2d15306-kube-api-access-27p54\") pod \"payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k\" (UID: \"11a54968-bcf4-4f36-97a7-86b3c2d15306\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:12.767851 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.767811    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:12.802968 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.802931    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nzp\" (UniqueName: \"kubernetes.io/projected/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-kube-api-access-n8nzp\") pod \"payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp\" (UID: \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:12.803179 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.803011    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp\" (UID: \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:12.904436 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.904356    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nzp\" (UniqueName: \"kubernetes.io/projected/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-kube-api-access-n8nzp\") pod \"payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp\" (UID: \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:12.904947 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.904446    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp\" (UID: \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:12.905482 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.905455    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp\" (UID: \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:12.965677 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:12.965632    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nzp\" (UniqueName: \"kubernetes.io/projected/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-kube-api-access-n8nzp\") pod \"payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp\" (UID: \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:13.157821 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:13.157791    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"]
Nov 01 06:00:13.244457 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:13.244428    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:13.616766 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:13.616741    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"]
Nov 01 06:00:13.618728 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:00:13.618705    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b4924e_d4f1_4e9d_b4ef_94ecdd8284c5.slice/crio-56e134a55c9d0bd97ddae1616a00b253d624dcd4242faa9633ec054ec8d02808 WatchSource:0}: Error finding container 56e134a55c9d0bd97ddae1616a00b253d624dcd4242faa9633ec054ec8d02808: Status 404 returned error can't find the container with id 56e134a55c9d0bd97ddae1616a00b253d624dcd4242faa9633ec054ec8d02808
Nov 01 06:00:14.011410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.011376    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" event={"ID":"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5","Type":"ContainerStarted","Data":"d79eeaa933ac865cccd78714a51c7e367d5f883c84badcf7b566f2c1f1a34a44"}
Nov 01 06:00:14.011864 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.011527    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:14.011864 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.011569    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" event={"ID":"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5","Type":"ContainerStarted","Data":"56e134a55c9d0bd97ddae1616a00b253d624dcd4242faa9633ec054ec8d02808"}
Nov 01 06:00:14.012960 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.012937    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" event={"ID":"11a54968-bcf4-4f36-97a7-86b3c2d15306","Type":"ContainerStarted","Data":"a39e7bea7c6a040b724d61ade48fc7f0991226687599ae699a3f6502af682c89"}
Nov 01 06:00:14.013073 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.012984    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:14.013073 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.013001    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" event={"ID":"11a54968-bcf4-4f36-97a7-86b3c2d15306","Type":"ContainerStarted","Data":"9c7f9623007f0dce78208da3745bf4cb6e17cc8c77da66e202969743dab82b20"}
Nov 01 06:00:14.131997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.131937    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" podStartSLOduration=2.131921339 podStartE2EDuration="2.131921339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:14.130082728 +0000 UTC m=+851.008056139" watchObservedRunningTime="2026-11-01 06:00:14.131921339 +0000 UTC m=+851.009894748"
Nov 01 06:00:14.229749 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.229712    5510 scope.go:122] "RemoveContainer" containerID="20026527bb233e0fc07c428e51b5012d8cf726ada9207bec22d8696f61107044"
Nov 01 06:00:14.229974 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.229774    5510 scope.go:122] "RemoveContainer" containerID="4f95a8ab699dcf913b20085f9cc6c7a4c2b68ff6507355c39cdb181f4ccd4063"
Nov 01 06:00:14.257840 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:14.251350    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" podStartSLOduration=2.251263344 podStartE2EDuration="2.251263344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:14.247970979 +0000 UTC m=+851.125944389" watchObservedRunningTime="2026-11-01 06:00:14.251263344 +0000 UTC m=+851.129236755"
Nov 01 06:00:15.021155 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:15.021119    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" event={"ID":"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870","Type":"ContainerStarted","Data":"6ab81b1bc29487628606574bf0c48b492239813add17ab43d3612f0f2b5be10c"}
Nov 01 06:00:15.022823 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:15.022799    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" event={"ID":"80f9f806-fc7c-4029-9e80-1058f4339e01","Type":"ContainerStarted","Data":"7e2b22d4207043ccfd4693f3438a3c31853deedf284e1f933dc5f1d567a10830"}
Nov 01 06:00:16.029645 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.029599    5510 generic.go:356] "Generic (PLEG): container finished" podID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerID="6ab81b1bc29487628606574bf0c48b492239813add17ab43d3612f0f2b5be10c" exitCode=60
Nov 01 06:00:16.030172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.029678    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" event={"ID":"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870","Type":"ContainerDied","Data":"6ab81b1bc29487628606574bf0c48b492239813add17ab43d3612f0f2b5be10c"}
Nov 01 06:00:16.030172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.029752    5510 scope.go:122] "RemoveContainer" containerID="20026527bb233e0fc07c428e51b5012d8cf726ada9207bec22d8696f61107044"
Nov 01 06:00:16.030172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.030106    5510 scope.go:122] "RemoveContainer" containerID="6ab81b1bc29487628606574bf0c48b492239813add17ab43d3612f0f2b5be10c"
Nov 01 06:00:16.030397 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:16.030373    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891880-g4gq2_odh-ai-gateway-infra(c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870"
Nov 01 06:00:16.031739 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.031722    5510 generic.go:356] "Generic (PLEG): container finished" podID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerID="7e2b22d4207043ccfd4693f3438a3c31853deedf284e1f933dc5f1d567a10830" exitCode=60
Nov 01 06:00:16.031854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.031794    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" event={"ID":"80f9f806-fc7c-4029-9e80-1058f4339e01","Type":"ContainerDied","Data":"7e2b22d4207043ccfd4693f3438a3c31853deedf284e1f933dc5f1d567a10830"}
Nov 01 06:00:16.032127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.032113    5510 scope.go:122] "RemoveContainer" containerID="7e2b22d4207043ccfd4693f3438a3c31853deedf284e1f933dc5f1d567a10830"
Nov 01 06:00:16.032341 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:16.032326    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g_odh-ai-gateway-infra(80f9f806-fc7c-4029-9e80-1058f4339e01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01"
Nov 01 06:00:16.041044 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.041024    5510 scope.go:122] "RemoveContainer" containerID="4f95a8ab699dcf913b20085f9cc6c7a4c2b68ff6507355c39cdb181f4ccd4063"
Nov 01 06:00:16.154950 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.154916    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"]
Nov 01 06:00:16.269451 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.269416    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"]
Nov 01 06:00:16.471184 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.471150    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"]
Nov 01 06:00:16.476099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.476081    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.529805 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.529778    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:00:16.532596 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.532571    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"]
Nov 01 06:00:16.546395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.546363    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.546523 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.546417    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxmf\" (UniqueName: \"kubernetes.io/projected/12180933-aa35-4b59-aa0b-4a72afe55d08-kube-api-access-7pxmf\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.546592 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.546573    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.647509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.647479    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.647680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.647541    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.647680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.647566    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxmf\" (UniqueName: \"kubernetes.io/projected/12180933-aa35-4b59-aa0b-4a72afe55d08-kube-api-access-7pxmf\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.648172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.648143    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.650246 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.650226    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.708851 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.708785    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxmf\" (UniqueName: \"kubernetes.io/projected/12180933-aa35-4b59-aa0b-4a72afe55d08-kube-api-access-7pxmf\") pod \"maas-api-revoke-sub-4a516248-hnmcs\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:16.789057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:16.788963    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:17.164961 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.164932    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"]
Nov 01 06:00:17.166639 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:00:17.166602    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12180933_aa35_4b59_aa0b_4a72afe55d08.slice/crio-8d95ad9dc701a189b9bd7f264d87aa9cb55d3b7449323ec05505416290199ac9 WatchSource:0}: Error finding container 8d95ad9dc701a189b9bd7f264d87aa9cb55d3b7449323ec05505416290199ac9: Status 404 returned error can't find the container with id 8d95ad9dc701a189b9bd7f264d87aa9cb55d3b7449323ec05505416290199ac9
Nov 01 06:00:17.174659 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.174633    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:17.203452 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.203430    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:17.238691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.238667    5510 scope.go:122] "RemoveContainer" containerID="c1ceb0e212fb8c66cfe03aaddab6afb380d255f0307c72572740c685d14cebd9"
Nov 01 06:00:17.254665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.254639    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-kube-api-access-wlkgg\" (UniqueName: \"kubernetes.io/projected/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-kube-api-access-wlkgg\") pod \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\" (UID: \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\") "
Nov 01 06:00:17.254776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.254753    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-maas-api-service-ca\") pod \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\" (UID: \"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870\") "
Nov 01 06:00:17.254883 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.254869    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f9f806-fc7c-4029-9e80-1058f4339e01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f9f806-fc7c-4029-9e80-1058f4339e01-maas-api-service-ca\") pod \"80f9f806-fc7c-4029-9e80-1058f4339e01\" (UID: \"80f9f806-fc7c-4029-9e80-1058f4339e01\") "
Nov 01 06:00:17.254964 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.254948    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f9f806-fc7c-4029-9e80-1058f4339e01-kube-api-access-6v7bx\" (UniqueName: \"kubernetes.io/projected/80f9f806-fc7c-4029-9e80-1058f4339e01-kube-api-access-6v7bx\") pod \"80f9f806-fc7c-4029-9e80-1058f4339e01\" (UID: \"80f9f806-fc7c-4029-9e80-1058f4339e01\") "
Nov 01 06:00:17.255146 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.255119    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-maas-api-service-ca" pod "c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" (UID: "c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:17.255230 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.255196    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f9f806-fc7c-4029-9e80-1058f4339e01-maas-api-service-ca" pod "80f9f806-fc7c-4029-9e80-1058f4339e01" (UID: "80f9f806-fc7c-4029-9e80-1058f4339e01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:17.255344 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.255322    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:17.255408 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.255350    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f9f806-fc7c-4029-9e80-1058f4339e01-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:17.256837 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.256805    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-kube-api-access-wlkgg" pod "c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" (UID: "c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870"). InnerVolumeSpecName "kube-api-access-wlkgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:17.257130 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.257109    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f9f806-fc7c-4029-9e80-1058f4339e01-kube-api-access-6v7bx" pod "80f9f806-fc7c-4029-9e80-1058f4339e01" (UID: "80f9f806-fc7c-4029-9e80-1058f4339e01"). InnerVolumeSpecName "kube-api-access-6v7bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:17.356406 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.356330    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v7bx\" (UniqueName: \"kubernetes.io/projected/80f9f806-fc7c-4029-9e80-1058f4339e01-kube-api-access-6v7bx\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:17.356406 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:17.356356    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlkgg\" (UniqueName: \"kubernetes.io/projected/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870-kube-api-access-wlkgg\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:17.545405 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:17.545365    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:00:18.047571 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.047534    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs" event={"ID":"12180933-aa35-4b59-aa0b-4a72afe55d08","Type":"ContainerStarted","Data":"75966cf707e54ddb5aab5fa88595e79e52154680ab485429a0e6967297c5e5ad"}
Nov 01 06:00:18.047748 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.047591    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs" event={"ID":"12180933-aa35-4b59-aa0b-4a72afe55d08","Type":"ContainerStarted","Data":"8d95ad9dc701a189b9bd7f264d87aa9cb55d3b7449323ec05505416290199ac9"}
Nov 01 06:00:18.049406 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.049379    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" event={"ID":"1556ac3f-d13e-47ed-b971-308e9cf90ce8","Type":"ContainerStarted","Data":"7e60d99a52ee557efa8f6bbf5e34c9de1888a6164358801a32d029e62968d0fd"}
Nov 01 06:00:18.050601 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.050585    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"
Nov 01 06:00:18.050691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.050613    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2" event={"ID":"c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870","Type":"ContainerDied","Data":"93e3db51f01897b2bffeb08c9643f4156417e85bc1ca0c52852a43792e697633"}
Nov 01 06:00:18.050691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.050674    5510 scope.go:122] "RemoveContainer" containerID="6ab81b1bc29487628606574bf0c48b492239813add17ab43d3612f0f2b5be10c"
Nov 01 06:00:18.051959 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.051943    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"
Nov 01 06:00:18.052018 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.051960    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g" event={"ID":"80f9f806-fc7c-4029-9e80-1058f4339e01","Type":"ContainerDied","Data":"d211701acab8edbd3b33eb7d75b5ef20c5cf6357b8f95b76ccda8203a5eb6d8e"}
Nov 01 06:00:18.061339 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.061324    5510 scope.go:122] "RemoveContainer" containerID="7e2b22d4207043ccfd4693f3438a3c31853deedf284e1f933dc5f1d567a10830"
Nov 01 06:00:18.166780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.166724    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs" podStartSLOduration=2.166711599 podStartE2EDuration="2.166711599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:18.164083603 +0000 UTC m=+855.042057013" watchObservedRunningTime="2026-11-01 06:00:18.166711599 +0000 UTC m=+855.044685009"
Nov 01 06:00:18.275894 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.275855    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"]
Nov 01 06:00:18.284401 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.284369    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d2807-29891880-sg78g"]
Nov 01 06:00:18.396795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.396697    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" podStartSLOduration=18.396682394 podStartE2EDuration="18.396682394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:18.396429346 +0000 UTC m=+855.274402757" watchObservedRunningTime="2026-11-01 06:00:18.396682394 +0000 UTC m=+855.274655801"
Nov 01 06:00:18.506747 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.506713    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"]
Nov 01 06:00:18.515015 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:18.514984    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-g4gq2"]
Nov 01 06:00:19.059661 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.059628    5510 generic.go:356] "Generic (PLEG): container finished" podID="12180933-aa35-4b59-aa0b-4a72afe55d08" containerID="75966cf707e54ddb5aab5fa88595e79e52154680ab485429a0e6967297c5e5ad" exitCode=0
Nov 01 06:00:19.059875 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.059714    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs" event={"ID":"12180933-aa35-4b59-aa0b-4a72afe55d08","Type":"ContainerDied","Data":"75966cf707e54ddb5aab5fa88595e79e52154680ab485429a0e6967297c5e5ad"}
Nov 01 06:00:19.061549 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.061526    5510 generic.go:356] "Generic (PLEG): container finished" podID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerID="7e60d99a52ee557efa8f6bbf5e34c9de1888a6164358801a32d029e62968d0fd" exitCode=60
Nov 01 06:00:19.061670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.061598    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" event={"ID":"1556ac3f-d13e-47ed-b971-308e9cf90ce8","Type":"ContainerDied","Data":"7e60d99a52ee557efa8f6bbf5e34c9de1888a6164358801a32d029e62968d0fd"}
Nov 01 06:00:19.061670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.061660    5510 scope.go:122] "RemoveContainer" containerID="c1ceb0e212fb8c66cfe03aaddab6afb380d255f0307c72572740c685d14cebd9"
Nov 01 06:00:19.062029 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.062015    5510 scope.go:122] "RemoveContainer" containerID="7e60d99a52ee557efa8f6bbf5e34c9de1888a6164358801a32d029e62968d0fd"
Nov 01 06:00:19.062318 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:19.062291    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs_odh-ai-gateway-infra(1556ac3f-d13e-47ed-b971-308e9cf90ce8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8"
Nov 01 06:00:19.240814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.240782    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" path="/var/lib/kubelet/pods/80f9f806-fc7c-4029-9e80-1058f4339e01/volumes"
Nov 01 06:00:19.241230 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.241195    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" path="/var/lib/kubelet/pods/c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870/volumes"
Nov 01 06:00:19.406036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:19.405958    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"]
Nov 01 06:00:20.216856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.216818    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:20.257290 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.257269    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:20.289115 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.289085    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1556ac3f-d13e-47ed-b971-308e9cf90ce8-kube-api-access-2plvm\" (UniqueName: \"kubernetes.io/projected/1556ac3f-d13e-47ed-b971-308e9cf90ce8-kube-api-access-2plvm\") pod \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\" (UID: \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\") "
Nov 01 06:00:20.289267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.289123    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-xks-ca\") pod \"12180933-aa35-4b59-aa0b-4a72afe55d08\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") "
Nov 01 06:00:20.289267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.289173    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1556ac3f-d13e-47ed-b971-308e9cf90ce8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1556ac3f-d13e-47ed-b971-308e9cf90ce8-maas-api-service-ca\") pod \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\" (UID: \"1556ac3f-d13e-47ed-b971-308e9cf90ce8\") "
Nov 01 06:00:20.289394 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.289331    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12180933-aa35-4b59-aa0b-4a72afe55d08-kube-api-access-7pxmf\" (UniqueName: \"kubernetes.io/projected/12180933-aa35-4b59-aa0b-4a72afe55d08-kube-api-access-7pxmf\") pod \"12180933-aa35-4b59-aa0b-4a72afe55d08\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") "
Nov 01 06:00:20.289473 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.289459    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-openshift-ca\") pod \"12180933-aa35-4b59-aa0b-4a72afe55d08\" (UID: \"12180933-aa35-4b59-aa0b-4a72afe55d08\") "
Nov 01 06:00:20.289575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.289555    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1556ac3f-d13e-47ed-b971-308e9cf90ce8-maas-api-service-ca" pod "1556ac3f-d13e-47ed-b971-308e9cf90ce8" (UID: "1556ac3f-d13e-47ed-b971-308e9cf90ce8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:20.289896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.289860    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-openshift-ca" pod "12180933-aa35-4b59-aa0b-4a72afe55d08" (UID: "12180933-aa35-4b59-aa0b-4a72afe55d08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:20.289896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.289876    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1556ac3f-d13e-47ed-b971-308e9cf90ce8-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:20.291420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.291400    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-xks-ca" pod "12180933-aa35-4b59-aa0b-4a72afe55d08" (UID: "12180933-aa35-4b59-aa0b-4a72afe55d08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:00:20.291509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.291480    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1556ac3f-d13e-47ed-b971-308e9cf90ce8-kube-api-access-2plvm" pod "1556ac3f-d13e-47ed-b971-308e9cf90ce8" (UID: "1556ac3f-d13e-47ed-b971-308e9cf90ce8"). InnerVolumeSpecName "kube-api-access-2plvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:20.291509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.291506    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12180933-aa35-4b59-aa0b-4a72afe55d08-kube-api-access-7pxmf" pod "12180933-aa35-4b59-aa0b-4a72afe55d08" (UID: "12180933-aa35-4b59-aa0b-4a72afe55d08"). InnerVolumeSpecName "kube-api-access-7pxmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:20.391445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.391363    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:20.391445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.391390    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2plvm\" (UniqueName: \"kubernetes.io/projected/1556ac3f-d13e-47ed-b971-308e9cf90ce8-kube-api-access-2plvm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:20.391445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.391400    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12180933-aa35-4b59-aa0b-4a72afe55d08-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:20.391445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:20.391409    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pxmf\" (UniqueName: \"kubernetes.io/projected/12180933-aa35-4b59-aa0b-4a72afe55d08-kube-api-access-7pxmf\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:21.077506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:21.077477    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"
Nov 01 06:00:21.077506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:21.077498    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs" event={"ID":"12180933-aa35-4b59-aa0b-4a72afe55d08","Type":"ContainerDied","Data":"8d95ad9dc701a189b9bd7f264d87aa9cb55d3b7449323ec05505416290199ac9"}
Nov 01 06:00:21.077782 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:21.077539    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d95ad9dc701a189b9bd7f264d87aa9cb55d3b7449323ec05505416290199ac9"
Nov 01 06:00:21.078816 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:21.078788    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs" event={"ID":"1556ac3f-d13e-47ed-b971-308e9cf90ce8","Type":"ContainerDied","Data":"3b59489e1e12f44599f25bb1019516eb05108a1d82d934473253e0cd01a2bc9b"}
Nov 01 06:00:21.078816 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:21.078808    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"
Nov 01 06:00:21.079021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:21.078866    5510 scope.go:122] "RemoveContainer" containerID="7e60d99a52ee557efa8f6bbf5e34c9de1888a6164358801a32d029e62968d0fd"
Nov 01 06:00:21.241415 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:21.241383    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"]
Nov 01 06:00:21.245059 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:21.245037    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891880-zgcjs"]
Nov 01 06:00:23.235300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:23.235271    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" path="/var/lib/kubelet/pods/1556ac3f-d13e-47ed-b971-308e9cf90ce8/volumes"
Nov 01 06:00:25.026057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:25.026030    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:00:25.026567 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:25.026109    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:00:35.742355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742317    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"]
Nov 01 06:00:35.742862 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742846    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:35.742906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742864    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:35.742906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742873    5510 cpu_manager.go:414] "containerMap: removing container" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.742906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742878    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.742906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742886    5510 cpu_manager.go:414] "containerMap: removing container" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.742906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742891    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.742906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742900    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:35.742906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742904    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742914    5510 cpu_manager.go:414] "containerMap: removing container" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742919    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742928    5510 cpu_manager.go:414] "containerMap: removing container" podUID="12180933-aa35-4b59-aa0b-4a72afe55d08" containerName="revoke-keys"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742935    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12180933-aa35-4b59-aa0b-4a72afe55d08" containerName="revoke-keys"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742951    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742956    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742977    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.742981    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.743074    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.743083    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.743090    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="12180933-aa35-4b59-aa0b-4a72afe55d08" containerName="revoke-keys"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.743101    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.743110    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.743117    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:35.743117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.743124    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.743529 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.743130    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f9f806-fc7c-4029-9e80-1058f4339e01" containerName="cleanup"
Nov 01 06:00:35.747762 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.747746    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:35.800893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.800845    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:00:35.803445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.803417    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"]
Nov 01 06:00:35.850383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.850346    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:00:35.850383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.850377    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:00:35.857391 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.857362    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:35.857506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.857430    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:35.857620 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.857596    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdbl\" (UniqueName: \"kubernetes.io/projected/92f907fb-c888-4e76-bc34-1de98d965b86-kube-api-access-smdbl\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:35.901255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.901228    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:00:35.958632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.958605    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:35.958808 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.958692    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdbl\" (UniqueName: \"kubernetes.io/projected/92f907fb-c888-4e76-bc34-1de98d965b86-kube-api-access-smdbl\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:35.958808 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.958752    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:35.959406 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.959383    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:35.961156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:35.961137    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:36.021581 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:36.021515    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdbl\" (UniqueName: \"kubernetes.io/projected/92f907fb-c888-4e76-bc34-1de98d965b86-kube-api-access-smdbl\") pod \"maas-api-revoke-sub-e4e23a5f-9ldnf\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:36.062501 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:36.062471    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:36.454275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:36.454251    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"]
Nov 01 06:00:36.455527 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:00:36.455503    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f907fb_c888_4e76_bc34_1de98d965b86.slice/crio-67f5c7d0d29ebcd8568796e7959de3552207f02cb7d5864efec981fbba340696 WatchSource:0}: Error finding container 67f5c7d0d29ebcd8568796e7959de3552207f02cb7d5864efec981fbba340696: Status 404 returned error can't find the container with id 67f5c7d0d29ebcd8568796e7959de3552207f02cb7d5864efec981fbba340696
Nov 01 06:00:37.182279 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:37.182189    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf" event={"ID":"92f907fb-c888-4e76-bc34-1de98d965b86","Type":"ContainerStarted","Data":"0eec449095604706540cb79a36d6f2c7f607b3cb510b2488954d4e4557679144"}
Nov 01 06:00:37.182279 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:37.182252    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf" event={"ID":"92f907fb-c888-4e76-bc34-1de98d965b86","Type":"ContainerStarted","Data":"67f5c7d0d29ebcd8568796e7959de3552207f02cb7d5864efec981fbba340696"}
Nov 01 06:00:37.299164 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:37.299107    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf" podStartSLOduration=2.2990939 podStartE2EDuration="2.2990939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:37.297450969 +0000 UTC m=+874.175424380" watchObservedRunningTime="2026-11-01 06:00:37.2990939 +0000 UTC m=+874.177067310"
Nov 01 06:00:38.189346 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:38.189315    5510 generic.go:356] "Generic (PLEG): container finished" podID="92f907fb-c888-4e76-bc34-1de98d965b86" containerID="0eec449095604706540cb79a36d6f2c7f607b3cb510b2488954d4e4557679144" exitCode=0
Nov 01 06:00:38.189737 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:38.189403    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf" event={"ID":"92f907fb-c888-4e76-bc34-1de98d965b86","Type":"ContainerDied","Data":"0eec449095604706540cb79a36d6f2c7f607b3cb510b2488954d4e4557679144"}
Nov 01 06:00:39.338061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.338039    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:39.498235 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.498206    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-xks-ca\") pod \"92f907fb-c888-4e76-bc34-1de98d965b86\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") "
Nov 01 06:00:39.498392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.498291    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-openshift-ca\") pod \"92f907fb-c888-4e76-bc34-1de98d965b86\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") "
Nov 01 06:00:39.498392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.498349    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f907fb-c888-4e76-bc34-1de98d965b86-kube-api-access-smdbl\" (UniqueName: \"kubernetes.io/projected/92f907fb-c888-4e76-bc34-1de98d965b86-kube-api-access-smdbl\") pod \"92f907fb-c888-4e76-bc34-1de98d965b86\" (UID: \"92f907fb-c888-4e76-bc34-1de98d965b86\") "
Nov 01 06:00:39.498682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.498652    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-openshift-ca" pod "92f907fb-c888-4e76-bc34-1de98d965b86" (UID: "92f907fb-c888-4e76-bc34-1de98d965b86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:39.500443 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.500416    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-xks-ca" pod "92f907fb-c888-4e76-bc34-1de98d965b86" (UID: "92f907fb-c888-4e76-bc34-1de98d965b86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:00:39.500574 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.500555    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f907fb-c888-4e76-bc34-1de98d965b86-kube-api-access-smdbl" pod "92f907fb-c888-4e76-bc34-1de98d965b86" (UID: "92f907fb-c888-4e76-bc34-1de98d965b86"). InnerVolumeSpecName "kube-api-access-smdbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:39.599892 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.599855    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:39.599892 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.599885    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f907fb-c888-4e76-bc34-1de98d965b86-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:39.599892 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:39.599900    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smdbl\" (UniqueName: \"kubernetes.io/projected/92f907fb-c888-4e76-bc34-1de98d965b86-kube-api-access-smdbl\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:40.201818 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:40.201787    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"
Nov 01 06:00:40.202022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:40.201792    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf" event={"ID":"92f907fb-c888-4e76-bc34-1de98d965b86","Type":"ContainerDied","Data":"67f5c7d0d29ebcd8568796e7959de3552207f02cb7d5864efec981fbba340696"}
Nov 01 06:00:40.202022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:40.201887    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f5c7d0d29ebcd8568796e7959de3552207f02cb7d5864efec981fbba340696"
Nov 01 06:00:41.191061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:41.191029    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:00:47.544481 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:00:47.544442    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:00:50.767684 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.767642    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"]
Nov 01 06:00:50.768382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768361    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:50.768475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768383    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:50.768475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768415    5510 cpu_manager.go:414] "containerMap: removing container" podUID="92f907fb-c888-4e76-bc34-1de98d965b86" containerName="revoke-keys"
Nov 01 06:00:50.768475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768423    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f907fb-c888-4e76-bc34-1de98d965b86" containerName="revoke-keys"
Nov 01 06:00:50.768475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768450    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:50.768475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768458    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:50.768726 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768576    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f907fb-c888-4e76-bc34-1de98d965b86" containerName="revoke-keys"
Nov 01 06:00:50.768726 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768590    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ebac6e-9cf5-4150-91ba-7b1b0b9f1870" containerName="cleanup"
Nov 01 06:00:50.768726 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.768605    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1556ac3f-d13e-47ed-b971-308e9cf90ce8" containerName="cleanup"
Nov 01 06:00:50.772100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.772080    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.814948 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.814915    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tls-certs\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.815123 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.814996    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-home\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.815123 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.815033    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q54n\" (UniqueName: \"kubernetes.io/projected/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-kube-api-access-7q54n\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.815123 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.815106    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-model-cache\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.815320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.815148    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-dshm\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.815320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.815216    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tmp-dir\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.843628 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.843602    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"distinct-w4-f73e89-kserve-self-signed-certs\""
Nov 01 06:00:50.846423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.846399    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"]
Nov 01 06:00:50.917913 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.917881    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tls-certs\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.918236 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.918212    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-home\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.918386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.918367    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q54n\" (UniqueName: \"kubernetes.io/projected/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-kube-api-access-7q54n\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.918514 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.918498    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-model-cache\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.918624 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.918610    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-dshm\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.925671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.918750    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tmp-dir\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.925671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.919524    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-model-cache\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.925671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.920169    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-home\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.925911 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.921074    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tmp-dir\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.927815 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.927760    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-dshm\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.933951 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.933902    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tls-certs\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:50.980686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:50.980657    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q54n\" (UniqueName: \"kubernetes.io/projected/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-kube-api-access-7q54n\") pod \"distinct-w4-f73e89-kserve-d6669785c-n9tkd\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:51.084071 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:51.083984    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:51.456415 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:51.456394    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"]
Nov 01 06:00:51.458381 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:00:51.458356    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84a010d6_0c1a_4a15_8af7_239ab2ecf8f3.slice/crio-110b4a07b0a66252b67f67597867d10a413e33e6f4a4dc9d3917cda39f19eab5 WatchSource:0}: Error finding container 110b4a07b0a66252b67f67597867d10a413e33e6f4a4dc9d3917cda39f19eab5: Status 404 returned error can't find the container with id 110b4a07b0a66252b67f67597867d10a413e33e6f4a4dc9d3917cda39f19eab5
Nov 01 06:00:52.268088 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:52.268044    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd" event={"ID":"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3","Type":"ContainerStarted","Data":"0fcd12be4cfb965aa9cae68c78fe780687b1109cf299ac4d71ef54e5e6e1c6f3"}
Nov 01 06:00:52.268487 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:52.268098    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd" event={"ID":"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3","Type":"ContainerStarted","Data":"110b4a07b0a66252b67f67597867d10a413e33e6f4a4dc9d3917cda39f19eab5"}
Nov 01 06:00:52.385200 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:52.385148    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd" podStartSLOduration=2.385134084 podStartE2EDuration="2.385134084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:52.383865629 +0000 UTC m=+889.261839041" watchObservedRunningTime="2026-11-01 06:00:52.385134084 +0000 UTC m=+889.263107491"
Nov 01 06:00:53.084452 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:53.084414    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:00:53.912106 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:53.912076    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"]
Nov 01 06:00:53.915971 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:53.915954    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:53.960163 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:53.960136    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:53.960274 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:53.960196    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dh2v\" (UniqueName: \"kubernetes.io/projected/6c25fc97-2a3d-4eee-a79a-a80822e21848-kube-api-access-7dh2v\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:53.960384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:53.960368    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:54.020395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.020351    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:00:54.020395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.020357    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:00:54.020395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.020394    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:00:54.020662 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.020416    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:00:54.022764 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.022741    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"]
Nov 01 06:00:54.061118 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.061079    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dh2v\" (UniqueName: \"kubernetes.io/projected/6c25fc97-2a3d-4eee-a79a-a80822e21848-kube-api-access-7dh2v\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:54.061272 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.061254    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:54.061334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.061304    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:54.061920 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.061897    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:54.063753 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.063736    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:54.121495 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.121464    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dh2v\" (UniqueName: \"kubernetes.io/projected/6c25fc97-2a3d-4eee-a79a-a80822e21848-kube-api-access-7dh2v\") pod \"maas-api-revoke-sub-79bb1c3d-wwt8h\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:54.230014 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.229978    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:54.606768 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:54.606744    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"]
Nov 01 06:00:54.608884 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:00:54.608860    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c25fc97_2a3d_4eee_a79a_a80822e21848.slice/crio-77b9d606a062d9615d5f4b3530045f0eaf83d1834dd2b8351b0e8d422179a0ed WatchSource:0}: Error finding container 77b9d606a062d9615d5f4b3530045f0eaf83d1834dd2b8351b0e8d422179a0ed: Status 404 returned error can't find the container with id 77b9d606a062d9615d5f4b3530045f0eaf83d1834dd2b8351b0e8d422179a0ed
Nov 01 06:00:55.297437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:55.297395    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h" event={"ID":"6c25fc97-2a3d-4eee-a79a-a80822e21848","Type":"ContainerStarted","Data":"b2d5e7421a1e58829bdc5a7a614966591ad0881224a33b25eb8ef3458638133b"}
Nov 01 06:00:55.297850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:55.297460    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h" event={"ID":"6c25fc97-2a3d-4eee-a79a-a80822e21848","Type":"ContainerStarted","Data":"77b9d606a062d9615d5f4b3530045f0eaf83d1834dd2b8351b0e8d422179a0ed"}
Nov 01 06:00:55.414275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:55.414211    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h" podStartSLOduration=2.414193215 podStartE2EDuration="2.414193215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:55.411410783 +0000 UTC m=+892.289384216" watchObservedRunningTime="2026-11-01 06:00:55.414193215 +0000 UTC m=+892.292166625"
Nov 01 06:00:56.303603 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:56.303571    5510 generic.go:356] "Generic (PLEG): container finished" podID="6c25fc97-2a3d-4eee-a79a-a80822e21848" containerID="b2d5e7421a1e58829bdc5a7a614966591ad0881224a33b25eb8ef3458638133b" exitCode=0
Nov 01 06:00:56.304020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:56.303629    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h" event={"ID":"6c25fc97-2a3d-4eee-a79a-a80822e21848","Type":"ContainerDied","Data":"b2d5e7421a1e58829bdc5a7a614966591ad0881224a33b25eb8ef3458638133b"}
Nov 01 06:00:56.447714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:56.447684    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:00:57.476248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.476218    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:57.515692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.515658    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-openshift-ca\") pod \"6c25fc97-2a3d-4eee-a79a-a80822e21848\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") "
Nov 01 06:00:57.515892 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.515872    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-xks-ca\") pod \"6c25fc97-2a3d-4eee-a79a-a80822e21848\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") "
Nov 01 06:00:57.515961 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.515938    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c25fc97-2a3d-4eee-a79a-a80822e21848-kube-api-access-7dh2v\" (UniqueName: \"kubernetes.io/projected/6c25fc97-2a3d-4eee-a79a-a80822e21848-kube-api-access-7dh2v\") pod \"6c25fc97-2a3d-4eee-a79a-a80822e21848\" (UID: \"6c25fc97-2a3d-4eee-a79a-a80822e21848\") "
Nov 01 06:00:57.516115 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.516085    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-openshift-ca" pod "6c25fc97-2a3d-4eee-a79a-a80822e21848" (UID: "6c25fc97-2a3d-4eee-a79a-a80822e21848"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:57.516396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.516369    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:57.518638 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.518613    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-xks-ca" pod "6c25fc97-2a3d-4eee-a79a-a80822e21848" (UID: "6c25fc97-2a3d-4eee-a79a-a80822e21848"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:00:57.518756 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.518660    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c25fc97-2a3d-4eee-a79a-a80822e21848-kube-api-access-7dh2v" pod "6c25fc97-2a3d-4eee-a79a-a80822e21848" (UID: "6c25fc97-2a3d-4eee-a79a-a80822e21848"). InnerVolumeSpecName "kube-api-access-7dh2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:57.617615 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.617535    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dh2v\" (UniqueName: \"kubernetes.io/projected/6c25fc97-2a3d-4eee-a79a-a80822e21848-kube-api-access-7dh2v\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:57.617615 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:57.617577    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c25fc97-2a3d-4eee-a79a-a80822e21848-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:58.318171 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:58.318143    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"
Nov 01 06:00:58.318171 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:58.318159    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h" event={"ID":"6c25fc97-2a3d-4eee-a79a-a80822e21848","Type":"ContainerDied","Data":"77b9d606a062d9615d5f4b3530045f0eaf83d1834dd2b8351b0e8d422179a0ed"}
Nov 01 06:00:58.318356 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:00:58.318200    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77b9d606a062d9615d5f4b3530045f0eaf83d1834dd2b8351b0e8d422179a0ed"
Nov 01 06:01:01.084324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:01.084292    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:01:01.096316 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:01.096292    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:01:01.345991 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:01.345910    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:01:06.013283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.013243    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"]
Nov 01 06:01:06.013994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.013973    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6c25fc97-2a3d-4eee-a79a-a80822e21848" containerName="revoke-keys"
Nov 01 06:01:06.013994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.013993    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c25fc97-2a3d-4eee-a79a-a80822e21848" containerName="revoke-keys"
Nov 01 06:01:06.014161 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.014149    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c25fc97-2a3d-4eee-a79a-a80822e21848" containerName="revoke-keys"
Nov 01 06:01:06.018814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.018793    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.108360 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.108324    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn42r\" (UniqueName: \"kubernetes.io/projected/78d0183d-aa4d-440c-87a4-32f9637d033b-kube-api-access-qn42r\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.108524 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.108380    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.108524 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.108445    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.121216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.121186    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:01:06.121216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.121212    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:01:06.121396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.121248    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:01:06.121396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.121275    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:01:06.123396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.123377    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"]
Nov 01 06:01:06.210022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.209994    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn42r\" (UniqueName: \"kubernetes.io/projected/78d0183d-aa4d-440c-87a4-32f9637d033b-kube-api-access-qn42r\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.210150 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.210036    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.210150 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.210056    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.210669 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.210651    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.212450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.212429    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.270501 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.270434    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn42r\" (UniqueName: \"kubernetes.io/projected/78d0183d-aa4d-440c-87a4-32f9637d033b-kube-api-access-qn42r\") pod \"maas-api-revoke-sub-6f36fcc4-fjbzh\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.331547 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.331512    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:06.705803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:06.705758    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"]
Nov 01 06:01:06.707969 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:01:06.707943    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d0183d_aa4d_440c_87a4_32f9637d033b.slice/crio-39a19c75d375a556fda6b47267b3b57b8fb96df462d31ebd345f8ba0458b4992 WatchSource:0}: Error finding container 39a19c75d375a556fda6b47267b3b57b8fb96df462d31ebd345f8ba0458b4992: Status 404 returned error can't find the container with id 39a19c75d375a556fda6b47267b3b57b8fb96df462d31ebd345f8ba0458b4992
Nov 01 06:01:07.371064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:07.370956    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh" event={"ID":"78d0183d-aa4d-440c-87a4-32f9637d033b","Type":"ContainerStarted","Data":"efc6b931dbc265cfc1c9db293d72097d841097fcd3b7cad8d18e6a22986abc8f"}
Nov 01 06:01:07.371064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:07.371034    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh" event={"ID":"78d0183d-aa4d-440c-87a4-32f9637d033b","Type":"ContainerStarted","Data":"39a19c75d375a556fda6b47267b3b57b8fb96df462d31ebd345f8ba0458b4992"}
Nov 01 06:01:07.486093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:07.486043    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh" podStartSLOduration=2.486029736 podStartE2EDuration="2.486029736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:07.485378143 +0000 UTC m=+904.363351554" watchObservedRunningTime="2026-11-01 06:01:07.486029736 +0000 UTC m=+904.364003146"
Nov 01 06:01:08.378016 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:08.377981    5510 generic.go:356] "Generic (PLEG): container finished" podID="78d0183d-aa4d-440c-87a4-32f9637d033b" containerID="efc6b931dbc265cfc1c9db293d72097d841097fcd3b7cad8d18e6a22986abc8f" exitCode=0
Nov 01 06:01:08.378400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:08.378064    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh" event={"ID":"78d0183d-aa4d-440c-87a4-32f9637d033b","Type":"ContainerDied","Data":"efc6b931dbc265cfc1c9db293d72097d841097fcd3b7cad8d18e6a22986abc8f"}
Nov 01 06:01:09.579017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.578989    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:09.648684 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.648648    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-openshift-ca\") pod \"78d0183d-aa4d-440c-87a4-32f9637d033b\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") "
Nov 01 06:01:09.648876 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.648739    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d0183d-aa4d-440c-87a4-32f9637d033b-kube-api-access-qn42r\" (UniqueName: \"kubernetes.io/projected/78d0183d-aa4d-440c-87a4-32f9637d033b-kube-api-access-qn42r\") pod \"78d0183d-aa4d-440c-87a4-32f9637d033b\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") "
Nov 01 06:01:09.648876 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.648802    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-xks-ca\") pod \"78d0183d-aa4d-440c-87a4-32f9637d033b\" (UID: \"78d0183d-aa4d-440c-87a4-32f9637d033b\") "
Nov 01 06:01:09.649038 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.649003    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-openshift-ca" pod "78d0183d-aa4d-440c-87a4-32f9637d033b" (UID: "78d0183d-aa4d-440c-87a4-32f9637d033b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:01:09.649218 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.649200    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:09.651038 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.651008    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-xks-ca" pod "78d0183d-aa4d-440c-87a4-32f9637d033b" (UID: "78d0183d-aa4d-440c-87a4-32f9637d033b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:01:09.651143 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.651075    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d0183d-aa4d-440c-87a4-32f9637d033b-kube-api-access-qn42r" pod "78d0183d-aa4d-440c-87a4-32f9637d033b" (UID: "78d0183d-aa4d-440c-87a4-32f9637d033b"). InnerVolumeSpecName "kube-api-access-qn42r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:09.750450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.750418    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn42r\" (UniqueName: \"kubernetes.io/projected/78d0183d-aa4d-440c-87a4-32f9637d033b-kube-api-access-qn42r\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:09.750450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:09.750454    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d0183d-aa4d-440c-87a4-32f9637d033b-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:10.418043 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:10.418010    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh" event={"ID":"78d0183d-aa4d-440c-87a4-32f9637d033b","Type":"ContainerDied","Data":"39a19c75d375a556fda6b47267b3b57b8fb96df462d31ebd345f8ba0458b4992"}
Nov 01 06:01:10.418043 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:10.418044    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"
Nov 01 06:01:10.418270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:10.418056    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39a19c75d375a556fda6b47267b3b57b8fb96df462d31ebd345f8ba0458b4992"
Nov 01 06:01:17.545088 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:01:17.545004    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:01:24.574193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.574156    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"]
Nov 01 06:01:24.574649 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.574636    5510 cpu_manager.go:414] "containerMap: removing container" podUID="78d0183d-aa4d-440c-87a4-32f9637d033b" containerName="revoke-keys"
Nov 01 06:01:24.574692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.574651    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d0183d-aa4d-440c-87a4-32f9637d033b" containerName="revoke-keys"
Nov 01 06:01:24.574740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.574731    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d0183d-aa4d-440c-87a4-32f9637d033b" containerName="revoke-keys"
Nov 01 06:01:24.578206 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.578192    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.637947 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.637554    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:01:24.640557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.640525    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"]
Nov 01 06:01:24.686699 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.686646    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:01:24.686699 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.686663    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:01:24.686699 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.686672    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:01:24.699197 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.699164    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.699324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.699216    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtz6\" (UniqueName: \"kubernetes.io/projected/541c7953-7054-4912-8f49-17488ca1e3bf-kube-api-access-cgtz6\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.699444 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.699326    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.800071 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.800038    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.800219 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.800078    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtz6\" (UniqueName: \"kubernetes.io/projected/541c7953-7054-4912-8f49-17488ca1e3bf-kube-api-access-cgtz6\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.800219 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.800119    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.800622 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.800603    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.802587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.802572    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.860801 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.860741    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtz6\" (UniqueName: \"kubernetes.io/projected/541c7953-7054-4912-8f49-17488ca1e3bf-kube-api-access-cgtz6\") pod \"maas-api-revoke-sub-9d23665f-nwl46\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:24.890464 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:24.890441    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:25.262238 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:25.262213    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"]
Nov 01 06:01:25.263958 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:01:25.263928    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541c7953_7054_4912_8f49_17488ca1e3bf.slice/crio-27cc73f59d81d8b4af0ace6bed6a3ff7ab025078389557b425ce8b6d8d46ab93 WatchSource:0}: Error finding container 27cc73f59d81d8b4af0ace6bed6a3ff7ab025078389557b425ce8b6d8d46ab93: Status 404 returned error can't find the container with id 27cc73f59d81d8b4af0ace6bed6a3ff7ab025078389557b425ce8b6d8d46ab93
Nov 01 06:01:25.499338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:25.499307    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46" event={"ID":"541c7953-7054-4912-8f49-17488ca1e3bf","Type":"ContainerStarted","Data":"27cc73f59d81d8b4af0ace6bed6a3ff7ab025078389557b425ce8b6d8d46ab93"}
Nov 01 06:01:26.506586 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:26.506553    5510 generic.go:356] "Generic (PLEG): container finished" podID="541c7953-7054-4912-8f49-17488ca1e3bf" containerID="5577b3ce8a47986b1fee0a9fe3dd6bfd7be74ab70feae933f7313582d6177129" exitCode=0
Nov 01 06:01:26.506998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:26.506635    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46" event={"ID":"541c7953-7054-4912-8f49-17488ca1e3bf","Type":"ContainerDied","Data":"5577b3ce8a47986b1fee0a9fe3dd6bfd7be74ab70feae933f7313582d6177129"}
Nov 01 06:01:27.663684 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.663665    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:27.832865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.832761    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541c7953-7054-4912-8f49-17488ca1e3bf-kube-api-access-cgtz6\" (UniqueName: \"kubernetes.io/projected/541c7953-7054-4912-8f49-17488ca1e3bf-kube-api-access-cgtz6\") pod \"541c7953-7054-4912-8f49-17488ca1e3bf\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") "
Nov 01 06:01:27.832865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.832815    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-openshift-ca\") pod \"541c7953-7054-4912-8f49-17488ca1e3bf\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") "
Nov 01 06:01:27.833052 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.832989    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-xks-ca\") pod \"541c7953-7054-4912-8f49-17488ca1e3bf\" (UID: \"541c7953-7054-4912-8f49-17488ca1e3bf\") "
Nov 01 06:01:27.833234 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.833211    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-openshift-ca" pod "541c7953-7054-4912-8f49-17488ca1e3bf" (UID: "541c7953-7054-4912-8f49-17488ca1e3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:01:27.833387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.833372    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:27.835055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.835017    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541c7953-7054-4912-8f49-17488ca1e3bf-kube-api-access-cgtz6" pod "541c7953-7054-4912-8f49-17488ca1e3bf" (UID: "541c7953-7054-4912-8f49-17488ca1e3bf"). InnerVolumeSpecName "kube-api-access-cgtz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:27.835055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.835037    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-xks-ca" pod "541c7953-7054-4912-8f49-17488ca1e3bf" (UID: "541c7953-7054-4912-8f49-17488ca1e3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:01:27.934022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.933985    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541c7953-7054-4912-8f49-17488ca1e3bf-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:27.934022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:27.934014    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtz6\" (UniqueName: \"kubernetes.io/projected/541c7953-7054-4912-8f49-17488ca1e3bf-kube-api-access-cgtz6\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:28.518599 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:28.518565    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"
Nov 01 06:01:28.518777 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:28.518617    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46" event={"ID":"541c7953-7054-4912-8f49-17488ca1e3bf","Type":"ContainerDied","Data":"27cc73f59d81d8b4af0ace6bed6a3ff7ab025078389557b425ce8b6d8d46ab93"}
Nov 01 06:01:28.518777 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:28.518663    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27cc73f59d81d8b4af0ace6bed6a3ff7ab025078389557b425ce8b6d8d46ab93"
Nov 01 06:01:34.043905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.043864    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"]
Nov 01 06:01:34.044584 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.044562    5510 cpu_manager.go:414] "containerMap: removing container" podUID="541c7953-7054-4912-8f49-17488ca1e3bf" containerName="revoke-keys"
Nov 01 06:01:34.044678 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.044586    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541c7953-7054-4912-8f49-17488ca1e3bf" containerName="revoke-keys"
Nov 01 06:01:34.044735 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.044719    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="541c7953-7054-4912-8f49-17488ca1e3bf" containerName="revoke-keys"
Nov 01 06:01:34.048036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.048015    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.100652 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.100621    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.100793 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.100669    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7nfw\" (UniqueName: \"kubernetes.io/projected/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-kube-api-access-h7nfw\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.100793 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.100762    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.151439 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.151405    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:01:34.151439 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.151414    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:01:34.151643 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.151447    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:01:34.151643 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.151421    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:01:34.153900 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.153880    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"]
Nov 01 06:01:34.201845 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.201806    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.202091 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.201870    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nfw\" (UniqueName: \"kubernetes.io/projected/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-kube-api-access-h7nfw\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.202091 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.201910    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.202560 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.202542    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.204326 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.204299    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.262908 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.262883    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7nfw\" (UniqueName: \"kubernetes.io/projected/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-kube-api-access-h7nfw\") pod \"maas-api-revoke-sub-d50c1f52-gtfkh\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.360768 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.360690    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:34.737779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:34.737751    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"]
Nov 01 06:01:34.739327 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:01:34.739289    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06be8e8c_591e_4f83_8b1c_15a0f38aa57f.slice/crio-729f28d246d113136c69ddef8df9b38c4cbf5297259163145ff621dc04bc68e1 WatchSource:0}: Error finding container 729f28d246d113136c69ddef8df9b38c4cbf5297259163145ff621dc04bc68e1: Status 404 returned error can't find the container with id 729f28d246d113136c69ddef8df9b38c4cbf5297259163145ff621dc04bc68e1
Nov 01 06:01:35.561391 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:35.561351    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh" event={"ID":"06be8e8c-591e-4f83-8b1c-15a0f38aa57f","Type":"ContainerStarted","Data":"5c586954a98d18213301a7d6c23012337b9d659e5cabe0ab980aa127239b26c0"}
Nov 01 06:01:35.561899 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:35.561399    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh" event={"ID":"06be8e8c-591e-4f83-8b1c-15a0f38aa57f","Type":"ContainerStarted","Data":"729f28d246d113136c69ddef8df9b38c4cbf5297259163145ff621dc04bc68e1"}
Nov 01 06:01:35.685141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:35.685089    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh" podStartSLOduration=1.685075814 podStartE2EDuration="1.685075814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:35.68264972 +0000 UTC m=+932.560623130" watchObservedRunningTime="2026-11-01 06:01:35.685075814 +0000 UTC m=+932.563049223"
Nov 01 06:01:36.568476 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:36.568445    5510 generic.go:356] "Generic (PLEG): container finished" podID="06be8e8c-591e-4f83-8b1c-15a0f38aa57f" containerID="5c586954a98d18213301a7d6c23012337b9d659e5cabe0ab980aa127239b26c0" exitCode=0
Nov 01 06:01:36.568892 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:36.568496    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh" event={"ID":"06be8e8c-591e-4f83-8b1c-15a0f38aa57f","Type":"ContainerDied","Data":"5c586954a98d18213301a7d6c23012337b9d659e5cabe0ab980aa127239b26c0"}
Nov 01 06:01:36.799818 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:36.799784    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:01:37.716618 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.716590    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:37.744744 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.744716    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-xks-ca\") pod \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") "
Nov 01 06:01:37.744898 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.744844    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-kube-api-access-h7nfw\" (UniqueName: \"kubernetes.io/projected/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-kube-api-access-h7nfw\") pod \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") "
Nov 01 06:01:37.744898 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.744887    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-openshift-ca\") pod \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\" (UID: \"06be8e8c-591e-4f83-8b1c-15a0f38aa57f\") "
Nov 01 06:01:37.745302 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.745273    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-openshift-ca" pod "06be8e8c-591e-4f83-8b1c-15a0f38aa57f" (UID: "06be8e8c-591e-4f83-8b1c-15a0f38aa57f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:01:37.745406 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.745385    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:37.747037 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.747013    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-kube-api-access-h7nfw" pod "06be8e8c-591e-4f83-8b1c-15a0f38aa57f" (UID: "06be8e8c-591e-4f83-8b1c-15a0f38aa57f"). InnerVolumeSpecName "kube-api-access-h7nfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:37.747258 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.747241    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-xks-ca" pod "06be8e8c-591e-4f83-8b1c-15a0f38aa57f" (UID: "06be8e8c-591e-4f83-8b1c-15a0f38aa57f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:01:37.846578 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.846493    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:37.846578 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:37.846530    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7nfw\" (UniqueName: \"kubernetes.io/projected/06be8e8c-591e-4f83-8b1c-15a0f38aa57f-kube-api-access-h7nfw\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:38.581240 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:38.581208    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"
Nov 01 06:01:38.581417 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:38.581237    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh" event={"ID":"06be8e8c-591e-4f83-8b1c-15a0f38aa57f","Type":"ContainerDied","Data":"729f28d246d113136c69ddef8df9b38c4cbf5297259163145ff621dc04bc68e1"}
Nov 01 06:01:38.581417 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:38.581285    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="729f28d246d113136c69ddef8df9b38c4cbf5297259163145ff621dc04bc68e1"
Nov 01 06:01:47.442544 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.442502    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"]
Nov 01 06:01:47.443213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.443192    5510 cpu_manager.go:414] "containerMap: removing container" podUID="06be8e8c-591e-4f83-8b1c-15a0f38aa57f" containerName="revoke-keys"
Nov 01 06:01:47.443303 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.443214    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06be8e8c-591e-4f83-8b1c-15a0f38aa57f" containerName="revoke-keys"
Nov 01 06:01:47.443385 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.443372    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="06be8e8c-591e-4f83-8b1c-15a0f38aa57f" containerName="revoke-keys"
Nov 01 06:01:47.446766 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.446745    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.545576 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:01:47.545543    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:01:47.551275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.551245    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cda04f\"/\"openshift-service-ca.crt\""
Nov 01 06:01:47.551275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.551257    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cda04f\"/\"default-dockercfg-287ld\""
Nov 01 06:01:47.551463 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.551279    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cda04f\"/\"kube-root-ca.crt\""
Nov 01 06:01:47.551463 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.551304    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cda04f\"/\"test-model-cda04f-kserve-self-signed-certs\""
Nov 01 06:01:47.553414 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.553396    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"]
Nov 01 06:01:47.567580 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.567557    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-model-cache\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.567644 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.567600    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tmp-dir\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.567689 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.567672    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-dshm\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.567763 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.567750    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qb9\" (UniqueName: \"kubernetes.io/projected/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-kube-api-access-s7qb9\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.567880 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.567863    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tls-certs\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.567919 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.567903    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-home\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669047 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669005    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-model-cache\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669066    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tmp-dir\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669104    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-dshm\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669180    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qb9\" (UniqueName: \"kubernetes.io/projected/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-kube-api-access-s7qb9\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669211    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tls-certs\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669237    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-home\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669513 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669462    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-model-cache\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669576 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669528    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tmp-dir\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.669713 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.669694    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-home\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.671432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.671412    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-dshm\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.671774 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.671757    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tls-certs\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.730231 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.730199    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qb9\" (UniqueName: \"kubernetes.io/projected/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-kube-api-access-s7qb9\") pod \"test-model-cda04f-kserve-54646dff59-55ksv\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") " pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:47.759175 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:47.759148    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:48.134315 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:48.134286    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"]
Nov 01 06:01:48.136536 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:01:48.136508    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab0e62f5_5531_4e7b_a7a2_4dd7270c3a3c.slice/crio-e59a442bf20ce6b4a24cf5ef5fbe89d6c479c5a479033aeba7673482b9a0606f WatchSource:0}: Error finding container e59a442bf20ce6b4a24cf5ef5fbe89d6c479c5a479033aeba7673482b9a0606f: Status 404 returned error can't find the container with id e59a442bf20ce6b4a24cf5ef5fbe89d6c479c5a479033aeba7673482b9a0606f
Nov 01 06:01:48.638281 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:48.638241    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv" event={"ID":"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c","Type":"ContainerStarted","Data":"5651d53f927f9dcb74891ab03f2fa3fc20253f4add284a4d7691fd7ccaa8a912"}
Nov 01 06:01:48.638281 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:48.638277    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv" event={"ID":"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c","Type":"ContainerStarted","Data":"e59a442bf20ce6b4a24cf5ef5fbe89d6c479c5a479033aeba7673482b9a0606f"}
Nov 01 06:01:48.758352 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:48.758292    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv" podStartSLOduration=1.75827717 podStartE2EDuration="1.75827717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:48.753660503 +0000 UTC m=+945.631633915" watchObservedRunningTime="2026-11-01 06:01:48.75827717 +0000 UTC m=+945.636250579"
Nov 01 06:01:48.759625 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:48.759602    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:50.054559 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.054528    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"]
Nov 01 06:01:50.059284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.059261    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.116244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.116198    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"]
Nov 01 06:01:50.162204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.162165    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:01:50.162397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.162244    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:01:50.162397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.162251    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:01:50.162397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.162310    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:01:50.202362 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.202322    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.202537 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.202443    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.202537 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.202500    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdzpt\" (UniqueName: \"kubernetes.io/projected/17c4fdb5-be6d-422f-8562-cc82868cc5bd-kube-api-access-kdzpt\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.304228 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.304192    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.304388 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.304276    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdzpt\" (UniqueName: \"kubernetes.io/projected/17c4fdb5-be6d-422f-8562-cc82868cc5bd-kube-api-access-kdzpt\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.304454 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.304392    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.305091 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.305033    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.307048 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.307029    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.365324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.365293    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdzpt\" (UniqueName: \"kubernetes.io/projected/17c4fdb5-be6d-422f-8562-cc82868cc5bd-kube-api-access-kdzpt\") pod \"maas-api-revoke-sub-824586e6-4qcmq\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.376106 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.376083    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:50.756795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:50.756775    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"]
Nov 01 06:01:50.758462 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:01:50.758433    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c4fdb5_be6d_422f_8562_cc82868cc5bd.slice/crio-39f8928bc872af57ece74960cd32c91bf7ae5ded0476981dfda8c119f1c5055b WatchSource:0}: Error finding container 39f8928bc872af57ece74960cd32c91bf7ae5ded0476981dfda8c119f1c5055b: Status 404 returned error can't find the container with id 39f8928bc872af57ece74960cd32c91bf7ae5ded0476981dfda8c119f1c5055b
Nov 01 06:01:51.661154 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:51.661088    5510 generic.go:356] "Generic (PLEG): container finished" podID="17c4fdb5-be6d-422f-8562-cc82868cc5bd" containerID="a5a8cbed9095bb1c117d36dc1b5bd494a0b7975fc80958bbf4f56b6245fbf1a3" exitCode=0
Nov 01 06:01:51.661508 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:51.661163    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq" event={"ID":"17c4fdb5-be6d-422f-8562-cc82868cc5bd","Type":"ContainerDied","Data":"a5a8cbed9095bb1c117d36dc1b5bd494a0b7975fc80958bbf4f56b6245fbf1a3"}
Nov 01 06:01:51.661508 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:51.661224    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq" event={"ID":"17c4fdb5-be6d-422f-8562-cc82868cc5bd","Type":"ContainerStarted","Data":"39f8928bc872af57ece74960cd32c91bf7ae5ded0476981dfda8c119f1c5055b"}
Nov 01 06:01:52.816000 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:52.815979    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:52.934499 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:52.934399    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-xks-ca\") pod \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") "
Nov 01 06:01:52.934499 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:52.934473    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-openshift-ca\") pod \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") "
Nov 01 06:01:52.934758 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:52.934545    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c4fdb5-be6d-422f-8562-cc82868cc5bd-kube-api-access-kdzpt\" (UniqueName: \"kubernetes.io/projected/17c4fdb5-be6d-422f-8562-cc82868cc5bd-kube-api-access-kdzpt\") pod \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\" (UID: \"17c4fdb5-be6d-422f-8562-cc82868cc5bd\") "
Nov 01 06:01:52.934935 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:52.934880    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-openshift-ca" pod "17c4fdb5-be6d-422f-8562-cc82868cc5bd" (UID: "17c4fdb5-be6d-422f-8562-cc82868cc5bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:01:52.935092 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:52.935003    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:52.937001 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:52.936976    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-xks-ca" pod "17c4fdb5-be6d-422f-8562-cc82868cc5bd" (UID: "17c4fdb5-be6d-422f-8562-cc82868cc5bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:01:52.937336 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:52.937302    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c4fdb5-be6d-422f-8562-cc82868cc5bd-kube-api-access-kdzpt" pod "17c4fdb5-be6d-422f-8562-cc82868cc5bd" (UID: "17c4fdb5-be6d-422f-8562-cc82868cc5bd"). InnerVolumeSpecName "kube-api-access-kdzpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:53.036197 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:53.036168    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdzpt\" (UniqueName: \"kubernetes.io/projected/17c4fdb5-be6d-422f-8562-cc82868cc5bd-kube-api-access-kdzpt\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:53.036197 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:53.036195    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c4fdb5-be6d-422f-8562-cc82868cc5bd-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:53.674470 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:53.674390    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"
Nov 01 06:01:53.674470 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:53.674423    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq" event={"ID":"17c4fdb5-be6d-422f-8562-cc82868cc5bd","Type":"ContainerDied","Data":"39f8928bc872af57ece74960cd32c91bf7ae5ded0476981dfda8c119f1c5055b"}
Nov 01 06:01:53.674660 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:53.674505    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39f8928bc872af57ece74960cd32c91bf7ae5ded0476981dfda8c119f1c5055b"
Nov 01 06:01:56.425641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:56.425605    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:01:57.713570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.713529    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"]
Nov 01 06:01:57.714101 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.714035    5510 cpu_manager.go:414] "containerMap: removing container" podUID="17c4fdb5-be6d-422f-8562-cc82868cc5bd" containerName="revoke-keys"
Nov 01 06:01:57.714101 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.714047    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c4fdb5-be6d-422f-8562-cc82868cc5bd" containerName="revoke-keys"
Nov 01 06:01:57.714225 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.714161    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c4fdb5-be6d-422f-8562-cc82868cc5bd" containerName="revoke-keys"
Nov 01 06:01:57.717741 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.717720    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:57.760162 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.760133    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:57.774320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.774298    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:57.777430 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.777400    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"]
Nov 01 06:01:57.822139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.822085    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cfe368\""
Nov 01 06:01:57.845562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.822463    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cfe368-dockercfg-sdsj7\""
Nov 01 06:01:57.845906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.823027    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7113c3d8-006a-4f56-8e09-7fc9a16ede18-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc\" (UID: \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:57.846025 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.845978    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtrss\" (UniqueName: \"kubernetes.io/projected/7113c3d8-006a-4f56-8e09-7fc9a16ede18-kube-api-access-mtrss\") pod \"payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc\" (UID: \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:57.875036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.875008    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"]
Nov 01 06:01:57.878896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.878870    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:57.925988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.925957    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"]
Nov 01 06:01:57.947282 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.947248    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7113c3d8-006a-4f56-8e09-7fc9a16ede18-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc\" (UID: \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:57.947440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.947290    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtrss\" (UniqueName: \"kubernetes.io/projected/7113c3d8-006a-4f56-8e09-7fc9a16ede18-kube-api-access-mtrss\") pod \"payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc\" (UID: \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:57.947440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.947320    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0555b2-a605-4691-80cd-cdf658ec4f25-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs\" (UID: \"9f0555b2-a605-4691-80cd-cdf658ec4f25\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:57.947440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.947370    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vp4\" (UniqueName: \"kubernetes.io/projected/9f0555b2-a605-4691-80cd-cdf658ec4f25-kube-api-access-67vp4\") pod \"payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs\" (UID: \"9f0555b2-a605-4691-80cd-cdf658ec4f25\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:57.948108 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:57.948085    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7113c3d8-006a-4f56-8e09-7fc9a16ede18-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc\" (UID: \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:58.008239 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.008168    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtrss\" (UniqueName: \"kubernetes.io/projected/7113c3d8-006a-4f56-8e09-7fc9a16ede18-kube-api-access-mtrss\") pod \"payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc\" (UID: \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:58.033069 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.033046    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:58.048653 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.048624    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vp4\" (UniqueName: \"kubernetes.io/projected/9f0555b2-a605-4691-80cd-cdf658ec4f25-kube-api-access-67vp4\") pod \"payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs\" (UID: \"9f0555b2-a605-4691-80cd-cdf658ec4f25\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:58.048814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.048799    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0555b2-a605-4691-80cd-cdf658ec4f25-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs\" (UID: \"9f0555b2-a605-4691-80cd-cdf658ec4f25\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:58.049480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.049463    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0555b2-a605-4691-80cd-cdf658ec4f25-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs\" (UID: \"9f0555b2-a605-4691-80cd-cdf658ec4f25\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:58.134779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.134745    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vp4\" (UniqueName: \"kubernetes.io/projected/9f0555b2-a605-4691-80cd-cdf658ec4f25-kube-api-access-67vp4\") pod \"payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs\" (UID: \"9f0555b2-a605-4691-80cd-cdf658ec4f25\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:58.193164 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.193136    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:58.408701 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.408677    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"]
Nov 01 06:01:58.410219 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:01:58.410190    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7113c3d8_006a_4f56_8e09_7fc9a16ede18.slice/crio-04c5f192f33ce232b35b9c05b6def23f4db465473073473f73c46dabb4027a2f WatchSource:0}: Error finding container 04c5f192f33ce232b35b9c05b6def23f4db465473073473f73c46dabb4027a2f: Status 404 returned error can't find the container with id 04c5f192f33ce232b35b9c05b6def23f4db465473073473f73c46dabb4027a2f
Nov 01 06:01:58.569757 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.569722    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"]
Nov 01 06:01:58.570186 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:01:58.570159    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0555b2_a605_4691_80cd_cdf658ec4f25.slice/crio-b53069dde74a94bc6b88c36b1d4b79150f3d6ab6bcf91c937efe4c707e7977d9 WatchSource:0}: Error finding container b53069dde74a94bc6b88c36b1d4b79150f3d6ab6bcf91c937efe4c707e7977d9: Status 404 returned error can't find the container with id b53069dde74a94bc6b88c36b1d4b79150f3d6ab6bcf91c937efe4c707e7977d9
Nov 01 06:01:58.704699 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.704623    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc" event={"ID":"7113c3d8-006a-4f56-8e09-7fc9a16ede18","Type":"ContainerStarted","Data":"2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821"}
Nov 01 06:01:58.704699 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.704671    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc" event={"ID":"7113c3d8-006a-4f56-8e09-7fc9a16ede18","Type":"ContainerStarted","Data":"04c5f192f33ce232b35b9c05b6def23f4db465473073473f73c46dabb4027a2f"}
Nov 01 06:01:58.704925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.704788    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:01:58.706398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.706373    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs" event={"ID":"9f0555b2-a605-4691-80cd-cdf658ec4f25","Type":"ContainerStarted","Data":"32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33"}
Nov 01 06:01:58.706496 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.706418    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs" event={"ID":"9f0555b2-a605-4691-80cd-cdf658ec4f25","Type":"ContainerStarted","Data":"b53069dde74a94bc6b88c36b1d4b79150f3d6ab6bcf91c937efe4c707e7977d9"}
Nov 01 06:01:58.706554 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.706524    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:01:58.717672 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.717654    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:01:58.824591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:58.824546    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc" podStartSLOduration=1.824533675 podStartE2EDuration="1.824533675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:58.822240622 +0000 UTC m=+955.700214032" watchObservedRunningTime="2026-11-01 06:01:58.824533675 +0000 UTC m=+955.702507085"
Nov 01 06:01:59.071620 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:01:59.071558    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs" podStartSLOduration=2.071541801 podStartE2EDuration="2.071541801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:59.070347673 +0000 UTC m=+955.948321085" watchObservedRunningTime="2026-11-01 06:01:59.071541801 +0000 UTC m=+955.949515216"
Nov 01 06:02:09.717026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:09.716991    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:02:09.717547 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:09.717075    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:02:15.770857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.770811    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"]
Nov 01 06:02:15.775814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.775799    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:15.830528 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.830476    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:02:15.844340 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.844307    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfk6p\" (UniqueName: \"kubernetes.io/projected/e69b2902-f3af-448c-bbcd-47fe3dfac911-kube-api-access-gfk6p\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:15.844472 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.844383    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:15.844522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.844479    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:15.879094 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.879061    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:02:15.879094 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.879094    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:02:15.879254 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.879128    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:02:15.881587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.881567    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"]
Nov 01 06:02:15.946035 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.946007    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfk6p\" (UniqueName: \"kubernetes.io/projected/e69b2902-f3af-448c-bbcd-47fe3dfac911-kube-api-access-gfk6p\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:15.946170 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.946050    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:15.946170 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.946094    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:15.946687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.946667    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:15.948440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:15.948422    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:16.006690 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:16.006667    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfk6p\" (UniqueName: \"kubernetes.io/projected/e69b2902-f3af-448c-bbcd-47fe3dfac911-kube-api-access-gfk6p\") pod \"maas-api-revoke-sub-f0c68063-dwppc\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:16.089881 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:16.089798    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:16.463869 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:16.463847    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"]
Nov 01 06:02:16.465498 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:02:16.465477    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69b2902_f3af_448c_bbcd_47fe3dfac911.slice/crio-8ce7081175197a0b6d936f4b69b43cce9d694edecf773183304971c56e8b58cc WatchSource:0}: Error finding container 8ce7081175197a0b6d936f4b69b43cce9d694edecf773183304971c56e8b58cc: Status 404 returned error can't find the container with id 8ce7081175197a0b6d936f4b69b43cce9d694edecf773183304971c56e8b58cc
Nov 01 06:02:16.846168 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:16.846129    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc" event={"ID":"e69b2902-f3af-448c-bbcd-47fe3dfac911","Type":"ContainerStarted","Data":"a46d3016c0a3a3b59ff0d7f10df3a941685d089921d6752973f36c9208fc97d2"}
Nov 01 06:02:16.846530 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:16.846219    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc" event={"ID":"e69b2902-f3af-448c-bbcd-47fe3dfac911","Type":"ContainerStarted","Data":"8ce7081175197a0b6d936f4b69b43cce9d694edecf773183304971c56e8b58cc"}
Nov 01 06:02:16.963107 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:16.963059    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc" podStartSLOduration=1.963045851 podStartE2EDuration="1.963045851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:16.961156274 +0000 UTC m=+973.839129680" watchObservedRunningTime="2026-11-01 06:02:16.963045851 +0000 UTC m=+973.841019260"
Nov 01 06:02:17.545345 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:02:17.545312    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:02:17.853998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:17.853964    5510 generic.go:356] "Generic (PLEG): container finished" podID="e69b2902-f3af-448c-bbcd-47fe3dfac911" containerID="a46d3016c0a3a3b59ff0d7f10df3a941685d089921d6752973f36c9208fc97d2" exitCode=0
Nov 01 06:02:17.854411 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:17.854047    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc" event={"ID":"e69b2902-f3af-448c-bbcd-47fe3dfac911","Type":"ContainerDied","Data":"a46d3016c0a3a3b59ff0d7f10df3a941685d089921d6752973f36c9208fc97d2"}
Nov 01 06:02:18.816157 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:18.816116    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:02:19.005302 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.005282    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:19.182565 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.182479    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-openshift-ca\") pod \"e69b2902-f3af-448c-bbcd-47fe3dfac911\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") "
Nov 01 06:02:19.182923 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.182892    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-xks-ca\") pod \"e69b2902-f3af-448c-bbcd-47fe3dfac911\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") "
Nov 01 06:02:19.183046 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.182951    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-openshift-ca" pod "e69b2902-f3af-448c-bbcd-47fe3dfac911" (UID: "e69b2902-f3af-448c-bbcd-47fe3dfac911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:02:19.183046 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.182990    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69b2902-f3af-448c-bbcd-47fe3dfac911-kube-api-access-gfk6p\" (UniqueName: \"kubernetes.io/projected/e69b2902-f3af-448c-bbcd-47fe3dfac911-kube-api-access-gfk6p\") pod \"e69b2902-f3af-448c-bbcd-47fe3dfac911\" (UID: \"e69b2902-f3af-448c-bbcd-47fe3dfac911\") "
Nov 01 06:02:19.183398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.183377    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:19.188069 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.187868    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-xks-ca" pod "e69b2902-f3af-448c-bbcd-47fe3dfac911" (UID: "e69b2902-f3af-448c-bbcd-47fe3dfac911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:02:19.188069 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.187934    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69b2902-f3af-448c-bbcd-47fe3dfac911-kube-api-access-gfk6p" pod "e69b2902-f3af-448c-bbcd-47fe3dfac911" (UID: "e69b2902-f3af-448c-bbcd-47fe3dfac911"). InnerVolumeSpecName "kube-api-access-gfk6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:02:19.284758 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.284719    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69b2902-f3af-448c-bbcd-47fe3dfac911-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:19.284758 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.284745    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfk6p\" (UniqueName: \"kubernetes.io/projected/e69b2902-f3af-448c-bbcd-47fe3dfac911-kube-api-access-gfk6p\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:19.866299 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.866262    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc" event={"ID":"e69b2902-f3af-448c-bbcd-47fe3dfac911","Type":"ContainerDied","Data":"8ce7081175197a0b6d936f4b69b43cce9d694edecf773183304971c56e8b58cc"}
Nov 01 06:02:19.866489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.866305    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"
Nov 01 06:02:19.866489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:19.866319    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce7081175197a0b6d936f4b69b43cce9d694edecf773183304971c56e8b58cc"
Nov 01 06:02:24.870394 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.870354    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"]
Nov 01 06:02:24.870973 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.870954    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e69b2902-f3af-448c-bbcd-47fe3dfac911" containerName="revoke-keys"
Nov 01 06:02:24.870973 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.870969    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69b2902-f3af-448c-bbcd-47fe3dfac911" containerName="revoke-keys"
Nov 01 06:02:24.871101 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.871079    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69b2902-f3af-448c-bbcd-47fe3dfac911" containerName="revoke-keys"
Nov 01 06:02:24.874587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.874566    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:24.929352 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.929308    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-cfe368-serving-cert\""
Nov 01 06:02:24.929513 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.929486    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:02:24.929586 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.929574    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:02:24.943642 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.943612    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20298942-580f-4a18-8b45-2ed53d03b46d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:24.943789 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.943663    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:24.943789 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.943745    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfzd\" (UniqueName: \"kubernetes.io/projected/20298942-580f-4a18-8b45-2ed53d03b46d-kube-api-access-8lfzd\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:24.943960 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.943879    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-metrics-certs\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:24.978588 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.978561    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 06:02:24.978716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.978583    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rqph5\""
Nov 01 06:02:24.981234 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:24.981213    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"]
Nov 01 06:02:25.044712 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:25.044679    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-metrics-certs\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:25.044922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:25.044757    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20298942-580f-4a18-8b45-2ed53d03b46d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:25.044922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:25.044806    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:25.044922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:25.044865    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfzd\" (UniqueName: \"kubernetes.io/projected/20298942-580f-4a18-8b45-2ed53d03b46d-kube-api-access-8lfzd\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:25.045113 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:02:25.044985    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-serving-cert: secret "maas-api-e2e-extmod-a-cfe368-serving-cert" not found
Nov 01 06:02:25.045113 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:02:25.045060    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls podName:20298942-580f-4a18-8b45-2ed53d03b46d nodeName:}" failed. No retries permitted until 2026-11-01 06:02:25.545043756 +0000 UTC m=+982.423017157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls") pod "maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27" (UID: "20298942-580f-4a18-8b45-2ed53d03b46d") : secret "maas-api-e2e-extmod-a-cfe368-serving-cert" not found
Nov 01 06:02:25.045467 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:25.045448    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20298942-580f-4a18-8b45-2ed53d03b46d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:25.047221 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:25.047203    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-metrics-certs\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:25.105697 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:25.105670    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfzd\" (UniqueName: \"kubernetes.io/projected/20298942-580f-4a18-8b45-2ed53d03b46d-kube-api-access-8lfzd\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:25.550803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:25.550772    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:25.550994 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:02:25.550926    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-serving-cert: secret "maas-api-e2e-extmod-a-cfe368-serving-cert" not found
Nov 01 06:02:25.551041 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:02:25.550996    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls podName:20298942-580f-4a18-8b45-2ed53d03b46d nodeName:}" failed. No retries permitted until 2026-11-01 06:02:26.550981451 +0000 UTC m=+983.428954839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls") pod "maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27" (UID: "20298942-580f-4a18-8b45-2ed53d03b46d") : secret "maas-api-e2e-extmod-a-cfe368-serving-cert" not found
Nov 01 06:02:26.558934 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:26.558901    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:26.561421 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:26.561401    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:26.687212 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:26.687182    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:27.072405 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:27.072379    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"]
Nov 01 06:02:27.074242 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:02:27.074210    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20298942_580f_4a18_8b45_2ed53d03b46d.slice/crio-4a2cbc3e5b331e185a24cfdb2b971af3873f4a84c06383b9e2c19eb985193b5c WatchSource:0}: Error finding container 4a2cbc3e5b331e185a24cfdb2b971af3873f4a84c06383b9e2c19eb985193b5c: Status 404 returned error can't find the container with id 4a2cbc3e5b331e185a24cfdb2b971af3873f4a84c06383b9e2c19eb985193b5c
Nov 01 06:02:27.075724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:27.075707    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:02:27.918333 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:27.918289    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27" event={"ID":"20298942-580f-4a18-8b45-2ed53d03b46d","Type":"ContainerStarted","Data":"4a2cbc3e5b331e185a24cfdb2b971af3873f4a84c06383b9e2c19eb985193b5c"}
Nov 01 06:02:29.933272 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:29.933238    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27" event={"ID":"20298942-580f-4a18-8b45-2ed53d03b46d","Type":"ContainerStarted","Data":"caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682"}
Nov 01 06:02:29.933665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:29.933370    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:30.053916 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:30.053865    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27" podStartSLOduration=4.103690211 podStartE2EDuration="6.053850014s" totalImagesPullingTime="1.950159803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:30.051452569 +0000 UTC m=+986.929425980" watchObservedRunningTime="2026-11-01 06:02:30.053850014 +0000 UTC m=+986.931823424"
Nov 01 06:02:35.944467 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:35.944432    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:02:36.190014 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.189984    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"]
Nov 01 06:02:36.194111 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.194092    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.248654 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.248628    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:02:36.249369 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.248704    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:02:36.252153 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.252130    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"]
Nov 01 06:02:36.269273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.269248    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsrd\" (UniqueName: \"kubernetes.io/projected/8f767baa-f942-4f0d-b3d6-4985b62e41b4-kube-api-access-6xsrd\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.269472 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.269446    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.269597 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.269578    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.370704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.370652    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsrd\" (UniqueName: \"kubernetes.io/projected/8f767baa-f942-4f0d-b3d6-4985b62e41b4-kube-api-access-6xsrd\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.375354 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.375322    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.375652 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.375632    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.385026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.384998    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.403871 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.389988    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.438775 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.438743    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsrd\" (UniqueName: \"kubernetes.io/projected/8f767baa-f942-4f0d-b3d6-4985b62e41b4-kube-api-access-6xsrd\") pod \"maas-api-revoke-sub-3a308883-5snqs\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.511446 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.511404    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:36.904629 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.904546    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"]
Nov 01 06:02:36.916583 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:02:36.915805    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f767baa_f942_4f0d_b3d6_4985b62e41b4.slice/crio-41279fdaa65dff87a0615b3b028653a201916810a631dd9de34ad32c63264ecd WatchSource:0}: Error finding container 41279fdaa65dff87a0615b3b028653a201916810a631dd9de34ad32c63264ecd: Status 404 returned error can't find the container with id 41279fdaa65dff87a0615b3b028653a201916810a631dd9de34ad32c63264ecd
Nov 01 06:02:36.977444 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:36.977408    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs" event={"ID":"8f767baa-f942-4f0d-b3d6-4985b62e41b4","Type":"ContainerStarted","Data":"41279fdaa65dff87a0615b3b028653a201916810a631dd9de34ad32c63264ecd"}
Nov 01 06:02:37.984780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:37.984692    5510 generic.go:356] "Generic (PLEG): container finished" podID="8f767baa-f942-4f0d-b3d6-4985b62e41b4" containerID="2c7331f98a8b7e54b8dd21aa4674e20e9eb19c430d969a3f8da701374f571bc6" exitCode=0
Nov 01 06:02:37.985186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:37.984787    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs" event={"ID":"8f767baa-f942-4f0d-b3d6-4985b62e41b4","Type":"ContainerDied","Data":"2c7331f98a8b7e54b8dd21aa4674e20e9eb19c430d969a3f8da701374f571bc6"}
Nov 01 06:02:39.138315 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.138293    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:39.210091 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.210068    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f767baa-f942-4f0d-b3d6-4985b62e41b4-kube-api-access-6xsrd\" (UniqueName: \"kubernetes.io/projected/8f767baa-f942-4f0d-b3d6-4985b62e41b4-kube-api-access-6xsrd\") pod \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") "
Nov 01 06:02:39.210214 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.210124    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-xks-ca\") pod \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") "
Nov 01 06:02:39.210214 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.210208    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-openshift-ca\") pod \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\" (UID: \"8f767baa-f942-4f0d-b3d6-4985b62e41b4\") "
Nov 01 06:02:39.210591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.210569    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-openshift-ca" pod "8f767baa-f942-4f0d-b3d6-4985b62e41b4" (UID: "8f767baa-f942-4f0d-b3d6-4985b62e41b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:02:39.212329 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.212309    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f767baa-f942-4f0d-b3d6-4985b62e41b4-kube-api-access-6xsrd" pod "8f767baa-f942-4f0d-b3d6-4985b62e41b4" (UID: "8f767baa-f942-4f0d-b3d6-4985b62e41b4"). InnerVolumeSpecName "kube-api-access-6xsrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:02:39.212329 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.212320    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-xks-ca" pod "8f767baa-f942-4f0d-b3d6-4985b62e41b4" (UID: "8f767baa-f942-4f0d-b3d6-4985b62e41b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:02:39.312038 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.312009    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsrd\" (UniqueName: \"kubernetes.io/projected/8f767baa-f942-4f0d-b3d6-4985b62e41b4-kube-api-access-6xsrd\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:39.312038 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.312034    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:39.312229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.312046    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f767baa-f942-4f0d-b3d6-4985b62e41b4-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:39.998406 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.998377    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"
Nov 01 06:02:39.998584 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.998382    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs" event={"ID":"8f767baa-f942-4f0d-b3d6-4985b62e41b4","Type":"ContainerDied","Data":"41279fdaa65dff87a0615b3b028653a201916810a631dd9de34ad32c63264ecd"}
Nov 01 06:02:39.998584 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:39.998499    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41279fdaa65dff87a0615b3b028653a201916810a631dd9de34ad32c63264ecd"
Nov 01 06:02:46.983667 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:46.983584    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"]
Nov 01 06:02:46.984255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:46.984199    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8f767baa-f942-4f0d-b3d6-4985b62e41b4" containerName="revoke-keys"
Nov 01 06:02:46.984255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:46.984212    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f767baa-f942-4f0d-b3d6-4985b62e41b4" containerName="revoke-keys"
Nov 01 06:02:46.984372 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:46.984285    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f767baa-f942-4f0d-b3d6-4985b62e41b4" containerName="revoke-keys"
Nov 01 06:02:46.987922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:46.987903    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.091149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.091118    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:02:47.091149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.091129    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:02:47.092652 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.092631    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.092707 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.092663    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.092813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.092793    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2hdt\" (UniqueName: \"kubernetes.io/projected/143b8d85-16f2-4da8-8f7b-f53367a06498-kube-api-access-n2hdt\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.095198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.095175    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"]
Nov 01 06:02:47.194388 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.194354    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.194388 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.194391    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.194607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.194445    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2hdt\" (UniqueName: \"kubernetes.io/projected/143b8d85-16f2-4da8-8f7b-f53367a06498-kube-api-access-n2hdt\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.195157 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.195138    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.196972 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.196950    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.255456 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.255393    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2hdt\" (UniqueName: \"kubernetes.io/projected/143b8d85-16f2-4da8-8f7b-f53367a06498-kube-api-access-n2hdt\") pod \"maas-api-revoke-sub-9ddccc16-vmx5n\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.302322 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.302301    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:47.544862 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:02:47.544763    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:02:47.679275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:47.679245    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"]
Nov 01 06:02:47.681031 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:02:47.681005    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143b8d85_16f2_4da8_8f7b_f53367a06498.slice/crio-d1e8dd76dc0de327ced084be5b7849b927ae5578e4de8425557cfdad5eb01634 WatchSource:0}: Error finding container d1e8dd76dc0de327ced084be5b7849b927ae5578e4de8425557cfdad5eb01634: Status 404 returned error can't find the container with id d1e8dd76dc0de327ced084be5b7849b927ae5578e4de8425557cfdad5eb01634
Nov 01 06:02:48.046936 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:48.046906    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n" event={"ID":"143b8d85-16f2-4da8-8f7b-f53367a06498","Type":"ContainerStarted","Data":"557bba4d40186e0a9f323f9bd3253881ba64dfbb2247bb2d1a40216ff364c0cf"}
Nov 01 06:02:48.047312 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:48.046954    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n" event={"ID":"143b8d85-16f2-4da8-8f7b-f53367a06498","Type":"ContainerStarted","Data":"d1e8dd76dc0de327ced084be5b7849b927ae5578e4de8425557cfdad5eb01634"}
Nov 01 06:02:48.163808 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:48.163713    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n" podStartSLOduration=2.163698906 podStartE2EDuration="2.163698906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:48.162384022 +0000 UTC m=+1005.040357426" watchObservedRunningTime="2026-11-01 06:02:48.163698906 +0000 UTC m=+1005.041672316"
Nov 01 06:02:49.055315 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:49.055276    5510 generic.go:356] "Generic (PLEG): container finished" podID="143b8d85-16f2-4da8-8f7b-f53367a06498" containerID="557bba4d40186e0a9f323f9bd3253881ba64dfbb2247bb2d1a40216ff364c0cf" exitCode=0
Nov 01 06:02:49.055315 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:49.055316    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n" event={"ID":"143b8d85-16f2-4da8-8f7b-f53367a06498","Type":"ContainerDied","Data":"557bba4d40186e0a9f323f9bd3253881ba64dfbb2247bb2d1a40216ff364c0cf"}
Nov 01 06:02:50.194901 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.194870    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:50.327862 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.327780    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-openshift-ca\") pod \"143b8d85-16f2-4da8-8f7b-f53367a06498\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") "
Nov 01 06:02:50.327998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.327875    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-xks-ca\") pod \"143b8d85-16f2-4da8-8f7b-f53367a06498\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") "
Nov 01 06:02:50.327998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.327911    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143b8d85-16f2-4da8-8f7b-f53367a06498-kube-api-access-n2hdt\" (UniqueName: \"kubernetes.io/projected/143b8d85-16f2-4da8-8f7b-f53367a06498-kube-api-access-n2hdt\") pod \"143b8d85-16f2-4da8-8f7b-f53367a06498\" (UID: \"143b8d85-16f2-4da8-8f7b-f53367a06498\") "
Nov 01 06:02:50.328182 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.328153    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-openshift-ca" pod "143b8d85-16f2-4da8-8f7b-f53367a06498" (UID: "143b8d85-16f2-4da8-8f7b-f53367a06498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:02:50.328556 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.328530    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:50.330221 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.330195    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-xks-ca" pod "143b8d85-16f2-4da8-8f7b-f53367a06498" (UID: "143b8d85-16f2-4da8-8f7b-f53367a06498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:02:50.330316 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.330261    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143b8d85-16f2-4da8-8f7b-f53367a06498-kube-api-access-n2hdt" pod "143b8d85-16f2-4da8-8f7b-f53367a06498" (UID: "143b8d85-16f2-4da8-8f7b-f53367a06498"). InnerVolumeSpecName "kube-api-access-n2hdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:02:50.430034 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.430006    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143b8d85-16f2-4da8-8f7b-f53367a06498-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:50.430034 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:50.430033    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2hdt\" (UniqueName: \"kubernetes.io/projected/143b8d85-16f2-4da8-8f7b-f53367a06498-kube-api-access-n2hdt\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:51.069177 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:51.069144    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n" event={"ID":"143b8d85-16f2-4da8-8f7b-f53367a06498","Type":"ContainerDied","Data":"d1e8dd76dc0de327ced084be5b7849b927ae5578e4de8425557cfdad5eb01634"}
Nov 01 06:02:51.069177 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:51.069160    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"
Nov 01 06:02:51.069379 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:51.069204    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1e8dd76dc0de327ced084be5b7849b927ae5578e4de8425557cfdad5eb01634"
Nov 01 06:02:52.701568 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:02:52.701531    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:03:03.225738 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.225699    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"]
Nov 01 06:03:03.226327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.226306    5510 cpu_manager.go:414] "containerMap: removing container" podUID="143b8d85-16f2-4da8-8f7b-f53367a06498" containerName="revoke-keys"
Nov 01 06:03:03.226392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.226330    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143b8d85-16f2-4da8-8f7b-f53367a06498" containerName="revoke-keys"
Nov 01 06:03:03.226503 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.226491    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="143b8d85-16f2-4da8-8f7b-f53367a06498" containerName="revoke-keys"
Nov 01 06:03:03.232576 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.232556    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285180 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285150    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nql5k\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-kube-api-access-nql5k\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285195    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285261    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285294    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285477 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285343    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285516 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285476    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab11984-22df-4438-afa2-a4bed63a76e5-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285573 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285559    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285620 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285590    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.285759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.285742    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.338503 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.338450    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"]
Nov 01 06:03:03.386814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.386778    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387034 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.386814    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387034 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.386878    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387034 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.386906    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387030    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab11984-22df-4438-afa2-a4bed63a76e5-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387125    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387163    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387379 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387221    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387379 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387247    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387379 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387254    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387364    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387383    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nql5k\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-kube-api-access-nql5k\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387594    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.387933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.387909    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab11984-22df-4438-afa2-a4bed63a76e5-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.389758 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.389734    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.389859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.389777    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.447986 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.447958    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.448246 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.448223    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nql5k\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-kube-api-access-nql5k\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.547790 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.547709    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:03.982582 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.982558    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"]
Nov 01 06:03:03.984178 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:03:03.984073    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab11984_22df_4438_afa2_a4bed63a76e5.slice/crio-b88cdc58a9a60ed77f6da71e9b4388e1f15078139f95b2d5c9a0827608f82c94 WatchSource:0}: Error finding container b88cdc58a9a60ed77f6da71e9b4388e1f15078139f95b2d5c9a0827608f82c94: Status 404 returned error can't find the container with id b88cdc58a9a60ed77f6da71e9b4388e1f15078139f95b2d5c9a0827608f82c94
Nov 01 06:03:03.986681 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.986647    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:03.986761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.986714    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:03.986761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:03.986743    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:04.150374 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:04.150333    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" event={"ID":"8ab11984-22df-4438-afa2-a4bed63a76e5","Type":"ContainerStarted","Data":"c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5"}
Nov 01 06:03:04.150585 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:04.150395    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" event={"ID":"8ab11984-22df-4438-afa2-a4bed63a76e5","Type":"ContainerStarted","Data":"b88cdc58a9a60ed77f6da71e9b4388e1f15078139f95b2d5c9a0827608f82c94"}
Nov 01 06:03:04.271721 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:04.271613    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podStartSLOduration=1.271593386 podStartE2EDuration="1.271593386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:04.269262994 +0000 UTC m=+1021.147236401" watchObservedRunningTime="2026-11-01 06:03:04.271593386 +0000 UTC m=+1021.149566798"
Nov 01 06:03:04.548166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:04.548077    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:05.550552 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:05.550520    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 06:03:05.550953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:05.550570    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 06:03:06.548452 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:06.548417    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 06:03:06.548648 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:06.548476    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 06:03:07.115097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.114646    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"]
Nov 01 06:03:07.124272 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.124238    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.144503 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144477    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istiod-ca-cert\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.144627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144513    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-envoy\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.144627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144547    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-token\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.144717 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144659    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-podinfo\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.144784 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144767    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmszf\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-kube-api-access-dmszf\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.144859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144799    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-data\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.144859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144818    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-socket\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.144980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144961    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-credential-socket\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.145017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.144984    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-certs\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.203766 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.203736    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"]
Nov 01 06:03:07.235931 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.235909    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3986bf-openshift-default-dockercfg-rsbx6\""
Nov 01 06:03:07.245653 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245605    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istiod-ca-cert\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.245730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245664    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-envoy\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.245730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245688    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-token\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.245730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245712    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-podinfo\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.245893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245774    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmszf\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-kube-api-access-dmszf\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.245893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245795    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-data\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.245893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245813    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-socket\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.245893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245868    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-credential-socket\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.245893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.245890    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-certs\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.246177 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.246148    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-certs\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.246330 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.246282    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-socket\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.246427 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.246376    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-data\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.246489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.246448    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-credential-socket\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.246588 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.246461    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istiod-ca-cert\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.248230 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.248212    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-envoy\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.248325 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.248307    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-podinfo\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.313884 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.312791    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-token\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.320776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.320748    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmszf\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-kube-api-access-dmszf\") pod \"e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") " pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.445104 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.443896    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:07.549030 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.548881    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 06:03:07.549030 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.548935    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 06:03:07.933602 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.933567    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"]
Nov 01 06:03:07.981094 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.980513    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:07.981094 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.980607    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:07.981094 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:07.980654    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:08.224694 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:08.223632    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" event={"ID":"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf","Type":"ContainerStarted","Data":"3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886"}
Nov 01 06:03:08.224694 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:08.223701    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" event={"ID":"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf","Type":"ContainerStarted","Data":"798bf0ec4d9f802e5cd3e6de0aa9a3b5f4bbdb07baa548c4d4ae763fe219258c"}
Nov 01 06:03:08.353035 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:08.352970    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podStartSLOduration=1.352952657 podStartE2EDuration="1.352952657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:08.352438457 +0000 UTC m=+1025.230411867" watchObservedRunningTime="2026-11-01 06:03:08.352952657 +0000 UTC m=+1025.230926070"
Nov 01 06:03:08.446294 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:08.446257    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:08.555904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:08.548170    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 06:03:08.555904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:08.548272    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 06:03:09.450516 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:09.450483    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:09.548582 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:09.548549    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 06:03:09.548763 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:09.548597    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 06:03:10.236464 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:10.236432    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:10.237525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:10.237504    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:03:10.548817 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:10.548724    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 06:03:10.548817 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:10.548782    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 06:03:11.620340 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:11.620308    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:11.620804 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:11.620780    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:11.621706 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:11.621684    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:03:13.514320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.514280    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"]
Nov 01 06:03:13.518412 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.518396    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.549869 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.549822    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.550044 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.549965    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpk5b\" (UniqueName: \"kubernetes.io/projected/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-kube-api-access-tpk5b\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.550044 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.550009    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.575496 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.575467    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"]
Nov 01 06:03:13.621142 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.621117    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:03:13.621295 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.621164    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:03:13.651040 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.651016    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.651165 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.651082    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpk5b\" (UniqueName: \"kubernetes.io/projected/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-kube-api-access-tpk5b\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.651165 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.651104    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.651647 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.651628    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.653623 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.653586    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.711674 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.711643    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpk5b\" (UniqueName: \"kubernetes.io/projected/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-kube-api-access-tpk5b\") pod \"maas-api-revoke-sub-3a31a638-9ph74\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:13.832186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:13.832109    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:14.208277 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.208248    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"]
Nov 01 06:03:14.209956 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:03:14.209931    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894ad3c4_0e49_41c0_ab22_63dbc8544fa0.slice/crio-c3f22f68b292b647288c496b763bf529c3208582da0b02eacdfe2c51031616ae WatchSource:0}: Error finding container c3f22f68b292b647288c496b763bf529c3208582da0b02eacdfe2c51031616ae: Status 404 returned error can't find the container with id c3f22f68b292b647288c496b763bf529c3208582da0b02eacdfe2c51031616ae
Nov 01 06:03:14.260866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.260841    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74" event={"ID":"894ad3c4-0e49-41c0-ab22-63dbc8544fa0","Type":"ContainerStarted","Data":"c3f22f68b292b647288c496b763bf529c3208582da0b02eacdfe2c51031616ae"}
Nov 01 06:03:14.779455 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.779380    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"]
Nov 01 06:03:14.783227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.783207    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:14.838497 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.838471    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3986bf\""
Nov 01 06:03:14.838497 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.838481    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3986bf-dockercfg-86284\""
Nov 01 06:03:14.842511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.842483    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"]
Nov 01 06:03:14.900730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.898674    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48ed9e9-7396-4f40-8074-01195481a5af-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv\" (UID: \"f48ed9e9-7396-4f40-8074-01195481a5af\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:14.900730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.898771    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xj4\" (UniqueName: \"kubernetes.io/projected/f48ed9e9-7396-4f40-8074-01195481a5af-kube-api-access-l5xj4\") pod \"payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv\" (UID: \"f48ed9e9-7396-4f40-8074-01195481a5af\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:14.964247 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.963590    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"]
Nov 01 06:03:14.968435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.968409    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:14.993770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.993742    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"]
Nov 01 06:03:14.999704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.999678    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xj4\" (UniqueName: \"kubernetes.io/projected/f48ed9e9-7396-4f40-8074-01195481a5af-kube-api-access-l5xj4\") pod \"payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv\" (UID: \"f48ed9e9-7396-4f40-8074-01195481a5af\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:14.999954 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:14.999934    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48ed9e9-7396-4f40-8074-01195481a5af-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv\" (UID: \"f48ed9e9-7396-4f40-8074-01195481a5af\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:15.000746 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.000724    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48ed9e9-7396-4f40-8074-01195481a5af-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv\" (UID: \"f48ed9e9-7396-4f40-8074-01195481a5af\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:15.060629 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.060549    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xj4\" (UniqueName: \"kubernetes.io/projected/f48ed9e9-7396-4f40-8074-01195481a5af-kube-api-access-l5xj4\") pod \"payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv\" (UID: \"f48ed9e9-7396-4f40-8074-01195481a5af\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:15.100388 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.100361    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:15.100700 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.100678    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp\" (UID: \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:15.100803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.100784    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjc55\" (UniqueName: \"kubernetes.io/projected/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-kube-api-access-xjc55\") pod \"payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp\" (UID: \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:15.201859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.201804    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp\" (UID: \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:15.202031 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.201916    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjc55\" (UniqueName: \"kubernetes.io/projected/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-kube-api-access-xjc55\") pod \"payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp\" (UID: \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:15.202749 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.202718    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp\" (UID: \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:15.262753 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.262719    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjc55\" (UniqueName: \"kubernetes.io/projected/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-kube-api-access-xjc55\") pod \"payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp\" (UID: \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:15.267951 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.267923    5510 generic.go:356] "Generic (PLEG): container finished" podID="894ad3c4-0e49-41c0-ab22-63dbc8544fa0" containerID="617c6823488c95d5b50891080bd81543f3a7e9798f7c7c00363e10540d2517e8" exitCode=0
Nov 01 06:03:15.268079 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.267998    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74" event={"ID":"894ad3c4-0e49-41c0-ab22-63dbc8544fa0","Type":"ContainerDied","Data":"617c6823488c95d5b50891080bd81543f3a7e9798f7c7c00363e10540d2517e8"}
Nov 01 06:03:15.284733 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.284708    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:15.496895 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.496868    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"]
Nov 01 06:03:15.498050 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:03:15.498024    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48ed9e9_7396_4f40_8074_01195481a5af.slice/crio-9c807a8336dda31e1b7b79ec4dd1073a2a90b1146cf3a2510e059b502b73143a WatchSource:0}: Error finding container 9c807a8336dda31e1b7b79ec4dd1073a2a90b1146cf3a2510e059b502b73143a: Status 404 returned error can't find the container with id 9c807a8336dda31e1b7b79ec4dd1073a2a90b1146cf3a2510e059b502b73143a
Nov 01 06:03:15.662008 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:15.661983    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"]
Nov 01 06:03:15.663008 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:03:15.662979    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d84a2e_e1e2_4107_ad0e_0e70d9175fc8.slice/crio-032cc35df92c47046a70aab67c059bdbac87d3b59418c745ced2b22c33fa7b4f WatchSource:0}: Error finding container 032cc35df92c47046a70aab67c059bdbac87d3b59418c745ced2b22c33fa7b4f: Status 404 returned error can't find the container with id 032cc35df92c47046a70aab67c059bdbac87d3b59418c745ced2b22c33fa7b4f
Nov 01 06:03:16.280085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.280054    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp" event={"ID":"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8","Type":"ContainerStarted","Data":"221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0"}
Nov 01 06:03:16.280526 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.280112    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp" event={"ID":"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8","Type":"ContainerStarted","Data":"032cc35df92c47046a70aab67c059bdbac87d3b59418c745ced2b22c33fa7b4f"}
Nov 01 06:03:16.280526 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.280258    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:16.281738 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.281709    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv" event={"ID":"f48ed9e9-7396-4f40-8074-01195481a5af","Type":"ContainerStarted","Data":"76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1"}
Nov 01 06:03:16.281883 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.281755    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:16.281883 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.281773    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv" event={"ID":"f48ed9e9-7396-4f40-8074-01195481a5af","Type":"ContainerStarted","Data":"9c807a8336dda31e1b7b79ec4dd1073a2a90b1146cf3a2510e059b502b73143a"}
Nov 01 06:03:16.397643 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.397572    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp" podStartSLOduration=2.397556866 podStartE2EDuration="2.397556866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:16.39571341 +0000 UTC m=+1033.273686824" watchObservedRunningTime="2026-11-01 06:03:16.397556866 +0000 UTC m=+1033.275530295"
Nov 01 06:03:16.436654 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.436630    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:16.530907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.530333    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-kube-api-access-tpk5b\" (UniqueName: \"kubernetes.io/projected/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-kube-api-access-tpk5b\") pod \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") "
Nov 01 06:03:16.530907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.530491    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-openshift-ca\") pod \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") "
Nov 01 06:03:16.530907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.530526    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-xks-ca\") pod \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\" (UID: \"894ad3c4-0e49-41c0-ab22-63dbc8544fa0\") "
Nov 01 06:03:16.543523 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.538378    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-openshift-ca" pod "894ad3c4-0e49-41c0-ab22-63dbc8544fa0" (UID: "894ad3c4-0e49-41c0-ab22-63dbc8544fa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:03:16.547303 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.546406    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv" podStartSLOduration=2.546372328 podStartE2EDuration="2.546372328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:16.516553458 +0000 UTC m=+1033.394526868" watchObservedRunningTime="2026-11-01 06:03:16.546372328 +0000 UTC m=+1033.424345809"
Nov 01 06:03:16.547303 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.546840    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-xks-ca" pod "894ad3c4-0e49-41c0-ab22-63dbc8544fa0" (UID: "894ad3c4-0e49-41c0-ab22-63dbc8544fa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:03:16.548487 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.548452    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-kube-api-access-tpk5b" pod "894ad3c4-0e49-41c0-ab22-63dbc8544fa0" (UID: "894ad3c4-0e49-41c0-ab22-63dbc8544fa0"). InnerVolumeSpecName "kube-api-access-tpk5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:03:16.632516 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.632327    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpk5b\" (UniqueName: \"kubernetes.io/projected/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-kube-api-access-tpk5b\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:16.632516 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.632388    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:16.632516 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:16.632410    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894ad3c4-0e49-41c0-ab22-63dbc8544fa0-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:17.289108 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:17.289078    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"
Nov 01 06:03:17.289484 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:17.289122    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74" event={"ID":"894ad3c4-0e49-41c0-ab22-63dbc8544fa0","Type":"ContainerDied","Data":"c3f22f68b292b647288c496b763bf529c3208582da0b02eacdfe2c51031616ae"}
Nov 01 06:03:17.289484 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:17.289165    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3f22f68b292b647288c496b763bf529c3208582da0b02eacdfe2c51031616ae"
Nov 01 06:03:17.514695 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:17.514610    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:03:17.544999 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:03:17.544969    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:03:19.795028 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.793335    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"]
Nov 01 06:03:19.796558 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.796525    5510 cpu_manager.go:414] "containerMap: removing container" podUID="894ad3c4-0e49-41c0-ab22-63dbc8544fa0" containerName="revoke-keys"
Nov 01 06:03:19.796558 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.796552    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894ad3c4-0e49-41c0-ab22-63dbc8544fa0" containerName="revoke-keys"
Nov 01 06:03:19.796760 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.796744    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="894ad3c4-0e49-41c0-ab22-63dbc8544fa0" containerName="revoke-keys"
Nov 01 06:03:19.800519 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.800497    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:19.835688 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.835660    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"]
Nov 01 06:03:19.881234 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.881210    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-58d523\""
Nov 01 06:03:19.881234 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.881214    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-58d523-dockercfg-6h5sw\""
Nov 01 06:03:19.982922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.982887    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f526277b-e4c7-495c-8cce-3e13066081bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89\" (UID: \"f526277b-e4c7-495c-8cce-3e13066081bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:19.983079 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:19.982940    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnw58\" (UniqueName: \"kubernetes.io/projected/f526277b-e4c7-495c-8cce-3e13066081bc-kube-api-access-tnw58\") pod \"payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89\" (UID: \"f526277b-e4c7-495c-8cce-3e13066081bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:20.083807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:20.083731    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f526277b-e4c7-495c-8cce-3e13066081bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89\" (UID: \"f526277b-e4c7-495c-8cce-3e13066081bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:20.083807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:20.083778    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnw58\" (UniqueName: \"kubernetes.io/projected/f526277b-e4c7-495c-8cce-3e13066081bc-kube-api-access-tnw58\") pod \"payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89\" (UID: \"f526277b-e4c7-495c-8cce-3e13066081bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:20.084447 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:20.084427    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f526277b-e4c7-495c-8cce-3e13066081bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89\" (UID: \"f526277b-e4c7-495c-8cce-3e13066081bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:20.144856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:20.144801    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnw58\" (UniqueName: \"kubernetes.io/projected/f526277b-e4c7-495c-8cce-3e13066081bc-kube-api-access-tnw58\") pod \"payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89\" (UID: \"f526277b-e4c7-495c-8cce-3e13066081bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:20.417256 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:20.417173    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:20.793645 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:03:20.793595    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf526277b_e4c7_495c_8cce_3e13066081bc.slice/crio-db570fcb3923a343fd9276c7b322d9d183ca8fca6394e84e6bde628431c1e63f WatchSource:0}: Error finding container db570fcb3923a343fd9276c7b322d9d183ca8fca6394e84e6bde628431c1e63f: Status 404 returned error can't find the container with id db570fcb3923a343fd9276c7b322d9d183ca8fca6394e84e6bde628431c1e63f
Nov 01 06:03:20.794423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:20.794393    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"]
Nov 01 06:03:21.382631 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:21.382592    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89" event={"ID":"f526277b-e4c7-495c-8cce-3e13066081bc","Type":"ContainerStarted","Data":"3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02"}
Nov 01 06:03:21.383052 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:21.382654    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89" event={"ID":"f526277b-e4c7-495c-8cce-3e13066081bc","Type":"ContainerStarted","Data":"db570fcb3923a343fd9276c7b322d9d183ca8fca6394e84e6bde628431c1e63f"}
Nov 01 06:03:21.383052 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:21.382683    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:21.500392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:21.500343    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89" podStartSLOduration=2.500330477 podStartE2EDuration="2.500330477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:21.498433573 +0000 UTC m=+1038.376406982" watchObservedRunningTime="2026-11-01 06:03:21.500330477 +0000 UTC m=+1038.378303887"
Nov 01 06:03:27.292511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:27.292483    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:03:27.292949 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:27.292551    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:03:32.392308 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:32.392278    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:03:38.848411 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.848372    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"]
Nov 01 06:03:38.853277 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.853250    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.905429 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.905390    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f73e89-openshift-default-dockercfg-tqlnq\""
Nov 01 06:03:38.913970 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.913939    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.914123 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.914055    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.914123 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.914087    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f176a81-71c4-47c4-922f-42b022dcd7bb-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.914123 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.914116    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.914262 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.914244    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.914300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.914279    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.914351 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.914313    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bkj\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-kube-api-access-75bkj\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.914389 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.914377    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.914450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.914437    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:38.958941 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:38.958899    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"]
Nov 01 06:03:39.015147 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015113    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015332 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015172    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015332 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015197    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f176a81-71c4-47c4-922f-42b022dcd7bb-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015332 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015225    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015332 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015273    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015548 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015465    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015548 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015537    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bkj\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-kube-api-access-75bkj\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015657 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015555    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015645    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015677    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015846 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015780    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.015997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.015975    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f176a81-71c4-47c4-922f-42b022dcd7bb-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.016174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.016136    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.016306 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.016189    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.017704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.017684    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.017841 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.017813    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.076163 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.076134    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.076300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.076212    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bkj\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-kube-api-access-75bkj\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.169104 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.169024    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:39.598498 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.598472    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"]
Nov 01 06:03:39.599507 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:03:39.599480    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f176a81_71c4_47c4_922f_42b022dcd7bb.slice/crio-03d3f23e4d86a400efad4534d920c4b81989f541ffe5cab77da11a7a062c9a74 WatchSource:0}: Error finding container 03d3f23e4d86a400efad4534d920c4b81989f541ffe5cab77da11a7a062c9a74: Status 404 returned error can't find the container with id 03d3f23e4d86a400efad4534d920c4b81989f541ffe5cab77da11a7a062c9a74
Nov 01 06:03:39.602030 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.601986    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:39.602103 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.602062    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:39.602103 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:39.602090    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:40.508293 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:40.508260    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" event={"ID":"2f176a81-71c4-47c4-922f-42b022dcd7bb","Type":"ContainerStarted","Data":"2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c"}
Nov 01 06:03:40.508691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:40.508309    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" event={"ID":"2f176a81-71c4-47c4-922f-42b022dcd7bb","Type":"ContainerStarted","Data":"03d3f23e4d86a400efad4534d920c4b81989f541ffe5cab77da11a7a062c9a74"}
Nov 01 06:03:40.698811 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:40.698754    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podStartSLOduration=2.698737098 podStartE2EDuration="2.698737098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:40.69540506 +0000 UTC m=+1057.573378471" watchObservedRunningTime="2026-11-01 06:03:40.698737098 +0000 UTC m=+1057.576710508"
Nov 01 06:03:41.181854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:41.173195    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:41.193853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:41.192952    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:03:41.193853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:41.193019    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 06:03:42.191854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:42.176270    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:03:42.191854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:42.176348    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 06:03:43.194221 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:43.194188    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:03:43.194623 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:43.194236    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 06:03:44.187853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:44.178230    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:03:44.187853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:44.178292    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 06:03:44.596690 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:44.596642    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:03:45.183855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:45.173207    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:03:45.183855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:45.173272    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 06:03:46.170085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:46.170045    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:03:46.170585 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:46.170118    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 06:03:47.061987 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:47.061944    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"]
Nov 01 06:03:47.110920 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:47.110882    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b59859b-d54nb"]
Nov 01 06:03:47.246368 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:47.246335    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96273ce-152f-4248-88bc-02d9dc6487ba" path="/var/lib/kubelet/pods/a96273ce-152f-4248-88bc-02d9dc6487ba/volumes"
Nov 01 06:03:47.544865 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:03:47.544800    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:03:48.212989 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:48.212950    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:03:48.213204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:48.213021    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:03:48.459416 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:48.459386    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:48.459803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:48.459770    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:48.460571 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:48.460555    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:03:52.917118 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:52.917090    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"]
Nov 01 06:03:52.921048 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:52.921029    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:52.978403 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:52.978378    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"]
Nov 01 06:03:53.008653 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.008626    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.008778 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.008679    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.008853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.008776    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-metrics-certs\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.008954 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.008932    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhgc\" (UniqueName: \"kubernetes.io/projected/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-kube-api-access-tmhgc\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.024088 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.024050    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-58d523-serving-cert\""
Nov 01 06:03:53.110088 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.110056    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.110398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.110144    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.110398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.110187    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-metrics-certs\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.110398 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:03:53.110220    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-serving-cert: secret "maas-api-e2e-ipp-a-58d523-serving-cert" not found
Nov 01 06:03:53.110398 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:03:53.110301    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls podName:c1dd4aa6-2f7a-4762-b9e9-6dcf10876409 nodeName:}" failed. No retries permitted until 2026-11-01 06:03:53.610281626 +0000 UTC m=+1070.488255018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls") pod "maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp" (UID: "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409") : secret "maas-api-e2e-ipp-a-58d523-serving-cert" not found
Nov 01 06:03:53.110398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.110322    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhgc\" (UniqueName: \"kubernetes.io/projected/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-kube-api-access-tmhgc\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.110914 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.110877    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.113534 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.113509    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-metrics-certs\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.196100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.195974    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhgc\" (UniqueName: \"kubernetes.io/projected/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-kube-api-access-tmhgc\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.618422 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.618393    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.621173 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.621149    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls\") pod \"maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.835633 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.835600    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:53.981722 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.981686    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"]
Nov 01 06:03:53.985704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:53.985688    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.023538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.023501    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.023538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.023540    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.023737 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.023697    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brw7z\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-kube-api-access-brw7z\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.023775 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.023752    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.023818 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.023795    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.023925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.023911    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.023978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.023966    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ba1edb-8e44-4567-8936-6808f39ee230-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.024047 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.024033    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.024079 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.024061    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ba1edb-8e44-4567-8936-6808f39ee230-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.075266 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.075230    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"]
Nov 01 06:03:54.125072 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125041    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125097    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125147    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ba1edb-8e44-4567-8936-6808f39ee230-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125167    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125188    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ba1edb-8e44-4567-8936-6808f39ee230-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125241    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125259    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125310    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brw7z\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-kube-api-access-brw7z\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125354    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125517    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125551    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125587    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.125985 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.125960    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.126271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.126213    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ba1edb-8e44-4567-8936-6808f39ee230-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.127775 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.127749    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.127893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.127852    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ba1edb-8e44-4567-8936-6808f39ee230-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.186260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.186188    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.186385 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.186273    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brw7z\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-kube-api-access-brw7z\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.216807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.216781    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"]
Nov 01 06:03:54.217794 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:03:54.217770    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dd4aa6_2f7a_4762_b9e9_6dcf10876409.slice/crio-449a44a93db97769f73b3e52b013944b943daaf272c61e97816b7cfa2c54c565 WatchSource:0}: Error finding container 449a44a93db97769f73b3e52b013944b943daaf272c61e97816b7cfa2c54c565: Status 404 returned error can't find the container with id 449a44a93db97769f73b3e52b013944b943daaf272c61e97816b7cfa2c54c565
Nov 01 06:03:54.300228 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.300194    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:54.607677 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.607636    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp" event={"ID":"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409","Type":"ContainerStarted","Data":"449a44a93db97769f73b3e52b013944b943daaf272c61e97816b7cfa2c54c565"}
Nov 01 06:03:54.690398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.690371    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"]
Nov 01 06:03:54.691381 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:03:54.691358    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ba1edb_8e44_4567_8936_6808f39ee230.slice/crio-928365e34cc32c36f29e959e95309db901aed9e27d3148a7eea2abb8516189ff WatchSource:0}: Error finding container 928365e34cc32c36f29e959e95309db901aed9e27d3148a7eea2abb8516189ff: Status 404 returned error can't find the container with id 928365e34cc32c36f29e959e95309db901aed9e27d3148a7eea2abb8516189ff
Nov 01 06:03:54.694019 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.693988    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:54.694077 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.694052    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:54.694114 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:54.694081    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:03:55.615701 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:55.615658    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp" event={"ID":"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409","Type":"ContainerStarted","Data":"80f16483da39f8804768b9078a6cf9a5b963414db75b96237aaaaeabfea17166"}
Nov 01 06:03:55.616505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:55.615722    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:03:55.617397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:55.617360    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" event={"ID":"10ba1edb-8e44-4567-8936-6808f39ee230","Type":"ContainerStarted","Data":"2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b"}
Nov 01 06:03:55.617514 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:55.617410    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" event={"ID":"10ba1edb-8e44-4567-8936-6808f39ee230","Type":"ContainerStarted","Data":"928365e34cc32c36f29e959e95309db901aed9e27d3148a7eea2abb8516189ff"}
Nov 01 06:03:55.734998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:55.734946    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp" podStartSLOduration=2.926064657 podStartE2EDuration="3.734930549s" totalImagesPullingTime="808.865892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:55.732199197 +0000 UTC m=+1072.610172608" watchObservedRunningTime="2026-11-01 06:03:55.734930549 +0000 UTC m=+1072.612903958"
Nov 01 06:03:55.873959 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:55.873816    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" podStartSLOduration=2.8737941019999997 podStartE2EDuration="2.873794102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:55.87026981 +0000 UTC m=+1072.748243259" watchObservedRunningTime="2026-11-01 06:03:55.873794102 +0000 UTC m=+1072.751767509"
Nov 01 06:03:56.301225 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:56.301191    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:03:56.302561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:56.302529    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:03:56.302710 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:56.302593    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 01 06:03:57.301702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:57.301668    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:03:57.302226 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:57.301723    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 01 06:03:58.307583 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:58.307535    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:03:58.308157 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:58.307614    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 01 06:03:59.301088 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:59.301049    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:03:59.301297 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:03:59.301121    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 01 06:04:00.301290 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:00.301255    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:04:00.301727 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:00.301309    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 01 06:04:01.301606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:01.301571    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:04:01.302049 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:01.301644    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 01 06:04:01.631081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:01.630998    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:04:02.305313 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:02.305279    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:04:02.305768 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:02.305748    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:04:02.306528 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:02.306512    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:04:11.277984 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:11.277942    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:04:11.278481 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:11.278005    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:04:12.794399 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:12.794326    5510 generic.go:356] "Generic (PLEG): container finished" podID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerID="3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886" exitCode=0
Nov 01 06:04:12.794802 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:12.794409    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" event={"ID":"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf","Type":"ContainerDied","Data":"3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886"}
Nov 01 06:04:12.794883 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:12.794845    5510 scope.go:122] "RemoveContainer" containerID="3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886"
Nov 01 06:04:12.796252 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:12.796225    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:04:12.796325 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:12.796293    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:04:12.796365 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:12.796326    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:04:13.803239 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:13.803154    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" event={"ID":"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf","Type":"ContainerStarted","Data":"a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1"}
Nov 01 06:04:14.445223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:14.445195    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:04:14.445467 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:14.445447    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:04:14.445529 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:14.445495    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 06:04:15.444881 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:15.444816    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:04:15.445278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:15.444903    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 06:04:16.445013 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:16.444973    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:04:16.445400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:16.445042    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 06:04:17.445248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:17.445217    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:04:17.445631 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:17.445283    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 06:04:17.563808 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:04:17.563734    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:04:18.445497 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:18.445467    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:04:18.445986 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:18.445533    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 06:04:19.449156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:19.449123    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:04:19.449618 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:19.449602    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:04:19.450378 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:19.450356    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:04:23.893745 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:23.893715    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:04:46.200278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:46.200244    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"]
Nov 01 06:04:47.546353 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:04:47.546312    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:04:51.200643 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:51.200583    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89" podUID="f526277b-e4c7-495c-8cce-3e13066081bc" containerName="payload-processing" containerID="cri-o://3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02" gracePeriod=25
Nov 01 06:04:51.466575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:51.466555    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:04:51.513412 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:51.513388    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f526277b-e4c7-495c-8cce-3e13066081bc-kube-api-access-tnw58\" (UniqueName: \"kubernetes.io/projected/f526277b-e4c7-495c-8cce-3e13066081bc-kube-api-access-tnw58\") pod \"f526277b-e4c7-495c-8cce-3e13066081bc\" (UID: \"f526277b-e4c7-495c-8cce-3e13066081bc\") "
Nov 01 06:04:51.513541 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:51.513484    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f526277b-e4c7-495c-8cce-3e13066081bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f526277b-e4c7-495c-8cce-3e13066081bc-plugins-config-volume\") pod \"f526277b-e4c7-495c-8cce-3e13066081bc\" (UID: \"f526277b-e4c7-495c-8cce-3e13066081bc\") "
Nov 01 06:04:51.513866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:51.513822    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f526277b-e4c7-495c-8cce-3e13066081bc-plugins-config-volume" pod "f526277b-e4c7-495c-8cce-3e13066081bc" (UID: "f526277b-e4c7-495c-8cce-3e13066081bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:04:51.515951 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:51.515925    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f526277b-e4c7-495c-8cce-3e13066081bc-kube-api-access-tnw58" pod "f526277b-e4c7-495c-8cce-3e13066081bc" (UID: "f526277b-e4c7-495c-8cce-3e13066081bc"). InnerVolumeSpecName "kube-api-access-tnw58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:51.615401 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:51.615363    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnw58\" (UniqueName: \"kubernetes.io/projected/f526277b-e4c7-495c-8cce-3e13066081bc-kube-api-access-tnw58\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:51.615401 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:51.615403    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f526277b-e4c7-495c-8cce-3e13066081bc-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:52.179891 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.179854    5510 generic.go:356] "Generic (PLEG): container finished" podID="f526277b-e4c7-495c-8cce-3e13066081bc" containerID="3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02" exitCode=0
Nov 01 06:04:52.180093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.179965    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"
Nov 01 06:04:52.180093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.179993    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89" event={"ID":"f526277b-e4c7-495c-8cce-3e13066081bc","Type":"ContainerDied","Data":"3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02"}
Nov 01 06:04:52.180093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.180049    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89" event={"ID":"f526277b-e4c7-495c-8cce-3e13066081bc","Type":"ContainerDied","Data":"db570fcb3923a343fd9276c7b322d9d183ca8fca6394e84e6bde628431c1e63f"}
Nov 01 06:04:52.180093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.180072    5510 scope.go:122] "RemoveContainer" containerID="3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02"
Nov 01 06:04:52.191916 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.191897    5510 scope.go:122] "RemoveContainer" containerID="3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02"
Nov 01 06:04:52.192222 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:04:52.192201    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02\": container with ID starting with 3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02 not found: ID does not exist" containerID="3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02"
Nov 01 06:04:52.192290 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.192230    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02"} err="failed to get container status \"3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02\": rpc error: code = NotFound desc = could not find container \"3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02\": container with ID starting with 3b4c71c522b5c0984358fadf7ed14f39e8ca5b7ef3d786ed1df9f19ce4456d02 not found: ID does not exist"
Nov 01 06:04:52.343814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.343778    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"]
Nov 01 06:04:52.393554 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:52.393521    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58d523-8bfdc8ff9-xwn89"]
Nov 01 06:04:53.236245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:53.236215    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f526277b-e4c7-495c-8cce-3e13066081bc" path="/var/lib/kubelet/pods/f526277b-e4c7-495c-8cce-3e13066081bc/volumes"
Nov 01 06:04:54.032519 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.032484    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"]
Nov 01 06:04:54.033483 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.033458    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f526277b-e4c7-495c-8cce-3e13066081bc" containerName="payload-processing"
Nov 01 06:04:54.033483 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.033483    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f526277b-e4c7-495c-8cce-3e13066081bc" containerName="payload-processing"
Nov 01 06:04:54.033666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.033654    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f526277b-e4c7-495c-8cce-3e13066081bc" containerName="payload-processing"
Nov 01 06:04:54.038922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.038891    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.057580 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.057557    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdcx\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-kube-api-access-bmdcx\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.057687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.057612    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.057740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.057708    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.057796 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.057740    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.057796 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.057769    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.057927 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.057910    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.057972 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.057958    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.058076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.058057    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.058131 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.058114    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.160001 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.159895    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdcx\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-kube-api-access-bmdcx\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.160427 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.160277    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.160695 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.160674    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.170185 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.170148    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.207225 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.207197    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.207881 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.207855    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.208471 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.208443    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.209161 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.171109    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.209269 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.207710    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.209269 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.202125    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"]
Nov 01 06:04:54.209269 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.208294    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.209432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.202454    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.209432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.208931    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.209597 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.209138    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.209805 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.209777    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.210306 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.210205    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.213493 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.213473    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.253070 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.253007    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdcx\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-kube-api-access-bmdcx\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.253994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.253944    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.352956 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.352878    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:54.780733 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.780703    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"]
Nov 01 06:04:54.781327 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:04:54.781297    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e2af86_bd15_4cd1_a1d1_3ec94b18c8c5.slice/crio-c0da1fedda8cbd7276a4c7ba167d8723c24e9af227b597c3a194c282fc4254f1 WatchSource:0}: Error finding container c0da1fedda8cbd7276a4c7ba167d8723c24e9af227b597c3a194c282fc4254f1: Status 404 returned error can't find the container with id c0da1fedda8cbd7276a4c7ba167d8723c24e9af227b597c3a194c282fc4254f1
Nov 01 06:04:54.783954 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.783913    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:04:54.784045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.783999    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:04:54.784045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:54.784038    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:04:55.224431 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:55.224395    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" event={"ID":"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5","Type":"ContainerStarted","Data":"88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9"}
Nov 01 06:04:55.224815 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:55.224457    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" event={"ID":"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5","Type":"ContainerStarted","Data":"c0da1fedda8cbd7276a4c7ba167d8723c24e9af227b597c3a194c282fc4254f1"}
Nov 01 06:04:55.345984 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:55.345930    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podStartSLOduration=2.345917055 podStartE2EDuration="2.345917055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:55.343389079 +0000 UTC m=+1132.221362514" watchObservedRunningTime="2026-11-01 06:04:55.345917055 +0000 UTC m=+1132.223890465"
Nov 01 06:04:55.353709 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:55.353675    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:04:55.355599 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:55.355576    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:04:55.355720 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:55.355624    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 01 06:04:55.903431 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:55.903399    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:04:56.364355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:56.364320    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:04:56.368606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:56.368567    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 01 06:04:57.353299 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:57.353269    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:04:57.353473 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:57.353318    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 01 06:04:58.353443 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:58.353413    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:04:58.353946 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:58.353474    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 01 06:04:59.353654 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:59.353611    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:04:59.354080 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:04:59.353684    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 01 06:05:00.353463 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:00.353431    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:05:00.353641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:00.353480    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 01 06:05:01.353857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:01.353811    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:05:01.354275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:01.353885    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 01 06:05:02.357810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:02.357779    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:05:02.358234 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:02.358085    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:05:02.358770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:02.358751    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:05:03.458797 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:03.458763    5510 scope.go:122] "RemoveContainer" containerID="56dd02096f738501c738e0ddda5b426f8c53a3ad2d883aac777bbb80c24f9459"
Nov 01 06:05:04.197590 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.197558    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"]
Nov 01 06:05:04.203323 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.203299    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.280904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.280871    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.281069 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.280925    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.281069 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.281033    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d1385-d230-4512-b014-6add6afeb772-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.281216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.281081    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d1385-d230-4512-b014-6add6afeb772-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.281216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.281183    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhvp\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-kube-api-access-hlhvp\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.281366 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.281304    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.281423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.281360    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.281423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.281391    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.281540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.281492    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.382773 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.382736    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.382981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.382784    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.382981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.382852    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.382981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.382912    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d1385-d230-4512-b014-6add6afeb772-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.382981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.382946    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d1385-d230-4512-b014-6add6afeb772-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.382991    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhvp\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-kube-api-access-hlhvp\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.383135    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.383187    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.383191    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.383214    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383541 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.383233    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.383560    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.383805    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d1385-d230-4512-b014-6add6afeb772-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.383859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.383819    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.385315 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.385283    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.385601 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.385583    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d1385-d230-4512-b014-6add6afeb772-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.417567 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.417525    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"]
Nov 01 06:05:04.445164 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.445129    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.445382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.445358    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhvp\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-kube-api-access-hlhvp\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.516919 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.516888    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:04.949381 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.949355    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"]
Nov 01 06:05:04.949934 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:05:04.949908    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49d1385_d230_4512_b014_6add6afeb772.slice/crio-7de5dd86a8dd229fd81b7d07019387a60f82c223fdbf6afc23a1e76d80aa87b4 WatchSource:0}: Error finding container 7de5dd86a8dd229fd81b7d07019387a60f82c223fdbf6afc23a1e76d80aa87b4: Status 404 returned error can't find the container with id 7de5dd86a8dd229fd81b7d07019387a60f82c223fdbf6afc23a1e76d80aa87b4
Nov 01 06:05:04.952399 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.952365    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:05:04.952467 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.952434    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:05:04.952508 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:04.952466    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:05:05.294715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:05.294652    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" event={"ID":"d49d1385-d230-4512-b014-6add6afeb772","Type":"ContainerStarted","Data":"521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446"}
Nov 01 06:05:05.294945 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:05.294738    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" event={"ID":"d49d1385-d230-4512-b014-6add6afeb772","Type":"ContainerStarted","Data":"7de5dd86a8dd229fd81b7d07019387a60f82c223fdbf6afc23a1e76d80aa87b4"}
Nov 01 06:05:05.415300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:05.415235    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podStartSLOduration=1.41521952 podStartE2EDuration="1.41521952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:05.414401074 +0000 UTC m=+1142.292374484" watchObservedRunningTime="2026-11-01 06:05:05.41521952 +0000 UTC m=+1142.293192929"
Nov 01 06:05:05.517662 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:05.517624    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:06.519041 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:06.519001    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 01 06:05:06.519432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:06.519063    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 01 06:05:07.517224 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:07.517193    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 01 06:05:07.517421 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:07.517245    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 01 06:05:08.517814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:08.517775    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 01 06:05:08.518255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:08.517874    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 01 06:05:09.517521 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:09.517487    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 01 06:05:09.517716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:09.517542    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 01 06:05:10.517996 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:10.517965    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 01 06:05:10.518387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:10.518015    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 01 06:05:11.518034 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:11.518001    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 01 06:05:11.518549 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:11.518067    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 01 06:05:12.542852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:12.542633    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:12.546850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:12.543693    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:12.546850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:12.545395    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:05:17.546806 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:05:17.546765    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:05:21.047161 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:21.047118    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"]
Nov 01 06:05:21.121226 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:21.113447    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a516248-hnmcs"]
Nov 01 06:05:21.236489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:21.236457    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12180933-aa35-4b59-aa0b-4a72afe55d08" path="/var/lib/kubelet/pods/12180933-aa35-4b59-aa0b-4a72afe55d08/volumes"
Nov 01 06:05:29.619411 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:29.619372    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:05:37.839908 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:37.839868    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"]
Nov 01 06:05:37.845108 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:37.845085    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:37.897558 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:37.897526    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2d3f8d-serving-cert\""
Nov 01 06:05:37.912578 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:37.912545    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwkt\" (UniqueName: \"kubernetes.io/projected/4d269181-c1ba-4620-a02b-eb3ac1c68da9-kube-api-access-mwwkt\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:37.912751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:37.912605    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:37.912898 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:37.912767    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:37.912898 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:37.912849    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d269181-c1ba-4620-a02b-eb3ac1c68da9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:37.950378 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:37.950341    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"]
Nov 01 06:05:38.013589 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.013560    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.013773 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.013648    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.013773 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.013685    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d269181-c1ba-4620-a02b-eb3ac1c68da9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.013773 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:05:38.013710    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-serving-cert: secret "maas-api-e2e-ipp-b-2d3f8d-serving-cert" not found
Nov 01 06:05:38.013773 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:05:38.013775    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls podName:4d269181-c1ba-4620-a02b-eb3ac1c68da9 nodeName:}" failed. No retries permitted until 2026-11-01 06:05:38.513756978 +0000 UTC m=+1175.391730370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls") pod "maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb" (UID: "4d269181-c1ba-4620-a02b-eb3ac1c68da9") : secret "maas-api-e2e-ipp-b-2d3f8d-serving-cert" not found
Nov 01 06:05:38.014014 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.013860    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwkt\" (UniqueName: \"kubernetes.io/projected/4d269181-c1ba-4620-a02b-eb3ac1c68da9-kube-api-access-mwwkt\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.014432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.014406    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d269181-c1ba-4620-a02b-eb3ac1c68da9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.016288 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.016268    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.077194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.077161    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwkt\" (UniqueName: \"kubernetes.io/projected/4d269181-c1ba-4620-a02b-eb3ac1c68da9-kube-api-access-mwwkt\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.523087 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.523056    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.525652 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.525623    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:38.760613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:38.760577    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:39.144410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:39.144381    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"]
Nov 01 06:05:39.145365 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:05:39.145338    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d269181_c1ba_4620_a02b_eb3ac1c68da9.slice/crio-0e9ba5b7e22cf2bf4cac69fd19e032b55e194ca14c48c04c1f891d7d7ae0a8ef WatchSource:0}: Error finding container 0e9ba5b7e22cf2bf4cac69fd19e032b55e194ca14c48c04c1f891d7d7ae0a8ef: Status 404 returned error can't find the container with id 0e9ba5b7e22cf2bf4cac69fd19e032b55e194ca14c48c04c1f891d7d7ae0a8ef
Nov 01 06:05:39.607243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:39.607208    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb" event={"ID":"4d269181-c1ba-4620-a02b-eb3ac1c68da9","Type":"ContainerStarted","Data":"0e9ba5b7e22cf2bf4cac69fd19e032b55e194ca14c48c04c1f891d7d7ae0a8ef"}
Nov 01 06:05:40.052479 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:40.052450    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"]
Nov 01 06:05:40.102022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:40.101958    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e23a5f-9ldnf"]
Nov 01 06:05:40.614759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:40.614719    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb" event={"ID":"4d269181-c1ba-4620-a02b-eb3ac1c68da9","Type":"ContainerStarted","Data":"49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778"}
Nov 01 06:05:40.615173 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:40.614910    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:40.734402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:40.734346    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb" podStartSLOduration=2.848030848 podStartE2EDuration="3.734331162s" totalImagesPullingTime="886.300314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:40.731414903 +0000 UTC m=+1177.609388314" watchObservedRunningTime="2026-11-01 06:05:40.734331162 +0000 UTC m=+1177.612304620"
Nov 01 06:05:41.236859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:41.236808    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f907fb-c888-4e76-bc34-1de98d965b86" path="/var/lib/kubelet/pods/92f907fb-c888-4e76-bc34-1de98d965b86/volumes"
Nov 01 06:05:46.626856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:46.626741    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:05:47.546372 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:05:47.546336    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:05:58.048173 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:58.048131    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"]
Nov 01 06:05:58.097412 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:58.097384    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79bb1c3d-wwt8h"]
Nov 01 06:05:59.237243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:05:59.237213    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c25fc97-2a3d-4eee-a79a-a80822e21848" path="/var/lib/kubelet/pods/6c25fc97-2a3d-4eee-a79a-a80822e21848/volumes"
Nov 01 06:06:00.538652 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.538619    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"]
Nov 01 06:06:00.542940 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.542921    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.595481 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.595445    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2d3f8d-openshift-default-dockercfg-zw2df\""
Nov 01 06:06:00.650112 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.650076    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"]
Nov 01 06:06:00.650780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.650748    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqq64\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-kube-api-access-kqq64\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.650934 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.650815    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.650934 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.650920    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.651064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.650982    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.651124 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.651096    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.651186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.651159    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.651248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.651205    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.651304 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.651290    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.651366 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.651308    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752114    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqq64\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-kube-api-access-kqq64\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752329 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752192    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752329 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752288    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752347    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752384    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752529 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752428    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752529 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752480    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752555    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752580    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752612    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752804 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752753    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.752894 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.752821    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.753036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.753012    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.753375 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.753352    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.755044 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.755022    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.755199 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.755180    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.813289 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.813215    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.813694 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.813660    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqq64\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-kube-api-access-kqq64\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:00.858563 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:00.858538    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:01.296566 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:01.296537    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"]
Nov 01 06:06:01.298312 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:06:01.298281    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7af0213_d788_40b0_8d4d_e68f0363f3d7.slice/crio-694cf75f11717cc75cc3ff0e1a4183fc77c0da2aaf89609dabb1848b52da7ea0 WatchSource:0}: Error finding container 694cf75f11717cc75cc3ff0e1a4183fc77c0da2aaf89609dabb1848b52da7ea0: Status 404 returned error can't find the container with id 694cf75f11717cc75cc3ff0e1a4183fc77c0da2aaf89609dabb1848b52da7ea0
Nov 01 06:06:01.300599 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:01.300564    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:06:01.300697 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:01.300627    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:06:01.300697 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:01.300658    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:06:01.768040 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:01.768006    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" event={"ID":"c7af0213-d788-40b0-8d4d-e68f0363f3d7","Type":"ContainerStarted","Data":"cf7d13c44235a47a57edc12548f7ebe8d10b0a97de822ed0613e65741c71a375"}
Nov 01 06:06:01.768429 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:01.768094    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" event={"ID":"c7af0213-d788-40b0-8d4d-e68f0363f3d7","Type":"ContainerStarted","Data":"694cf75f11717cc75cc3ff0e1a4183fc77c0da2aaf89609dabb1848b52da7ea0"}
Nov 01 06:06:01.858931 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:01.858889    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:02.860470 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:02.860431    5510 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:06:02.860864 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:02.860486    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 06:06:03.859339 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:03.859312    5510 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:06:03.859514 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:03.859358    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 06:06:04.859559 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:04.859525    5510 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:06:04.859945 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:04.859582    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 06:06:05.859494 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:05.859464    5510 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:06:05.859680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:05.859517    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 06:06:06.859477 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:06.859437    5510 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:06:06.859646 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:06.859505    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 06:06:07.863800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:07.863768    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:07.864196 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:07.864037    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:07.864785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:07.864767    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:06:08.036713 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:08.036664    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" podStartSLOduration=8.036650528 podStartE2EDuration="8.036650528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:01.893125432 +0000 UTC m=+1198.771098843" watchObservedRunningTime="2026-11-01 06:06:08.036650528 +0000 UTC m=+1204.914623962"
Nov 01 06:06:09.763274 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:09.763236    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:06:10.047165 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:10.047092    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"]
Nov 01 06:06:10.096627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:10.096596    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f36fcc4-fjbzh"]
Nov 01 06:06:11.257853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:11.256814    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d0183d-aa4d-440c-87a4-32f9637d033b" path="/var/lib/kubelet/pods/78d0183d-aa4d-440c-87a4-32f9637d033b/volumes"
Nov 01 06:06:17.545652 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:06:17.545608    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:06:28.046334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:28.046297    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"]
Nov 01 06:06:28.096183 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:28.096151    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d23665f-nwl46"]
Nov 01 06:06:29.236419 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:29.236387    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541c7953-7054-4912-8f49-17488ca1e3bf" path="/var/lib/kubelet/pods/541c7953-7054-4912-8f49-17488ca1e3bf/volumes"
Nov 01 06:06:38.061649 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:38.059658    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"]
Nov 01 06:06:38.101311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:38.101275    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d50c1f52-gtfkh"]
Nov 01 06:06:39.272024 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:39.271629    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06be8e8c-591e-4f83-8b1c-15a0f38aa57f" path="/var/lib/kubelet/pods/06be8e8c-591e-4f83-8b1c-15a0f38aa57f/volumes"
Nov 01 06:06:47.545080 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:06:47.545046    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:06:52.162992 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:52.162956    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:06:52.837886 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:52.813646    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"]
Nov 01 06:06:53.049425 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:53.048538    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"]
Nov 01 06:06:53.052443 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:53.052414    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824586e6-4qcmq"]
Nov 01 06:06:53.288453 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:06:53.288396    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c4fdb5-be6d-422f-8562-cc82868cc5bd" path="/var/lib/kubelet/pods/17c4fdb5-be6d-422f-8562-cc82868cc5bd/volumes"
Nov 01 06:07:00.120396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:00.120339    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"]
Nov 01 06:07:03.506903 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:03.506874    5510 scope.go:122] "RemoveContainer" containerID="75966cf707e54ddb5aab5fa88595e79e52154680ab485429a0e6967297c5e5ad"
Nov 01 06:07:03.517601 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:03.517581    5510 scope.go:122] "RemoveContainer" containerID="b2d5e7421a1e58829bdc5a7a614966591ad0881224a33b25eb8ef3458638133b"
Nov 01 06:07:03.526243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:03.526223    5510 scope.go:122] "RemoveContainer" containerID="0eec449095604706540cb79a36d6f2c7f607b3cb510b2488954d4e4557679144"
Nov 01 06:07:05.122077 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:05.122016    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw" podUID="f22f3138-7e24-4e67-a572-10e9e1d2a8e9" containerName="payload-pre-processing" containerID="cri-o://848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4" gracePeriod=25
Nov 01 06:07:07.918865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:07.918799    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv" podUID="ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" containerName="main" containerID="cri-o://5651d53f927f9dcb74891ab03f2fa3fc20253f4add284a4d7691fd7ccaa8a912" gracePeriod=45
Nov 01 06:07:08.584770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:08.584726    5510 generic.go:356] "Generic (PLEG): container finished" podID="ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" containerID="5651d53f927f9dcb74891ab03f2fa3fc20253f4add284a4d7691fd7ccaa8a912" exitCode=0
Nov 01 06:07:08.585121 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:08.584881    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv" event={"ID":"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c","Type":"ContainerDied","Data":"5651d53f927f9dcb74891ab03f2fa3fc20253f4add284a4d7691fd7ccaa8a912"}
Nov 01 06:07:08.628198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:08.628051    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw" podUID="f22f3138-7e24-4e67-a572-10e9e1d2a8e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:07:09.301371 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.301170    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:07:09.440095 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.440062    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tls-certs\") pod \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") "
Nov 01 06:07:09.441123 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.441101    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-home\") pod \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") "
Nov 01 06:07:09.445054 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.445034    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-dshm\") pod \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") "
Nov 01 06:07:09.445579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.445561    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tmp-dir\") pod \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") "
Nov 01 06:07:09.525085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.444948    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-home" pod "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" (UID: "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:09.525085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.451368    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tmp-dir" pod "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" (UID: "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:09.525085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.509988    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tls-certs" pod "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" (UID: "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:07:09.525085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.524663    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-model-cache" pod "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" (UID: "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:09.525085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.524705    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-model-cache\") pod \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") "
Nov 01 06:07:09.525085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.524856    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-kube-api-access-s7qb9\" (UniqueName: \"kubernetes.io/projected/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-kube-api-access-s7qb9\") pod \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\" (UID: \"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c\") "
Nov 01 06:07:09.532868 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.530532    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tmp-dir\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:09.532868 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.530559    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-model-cache\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:09.532868 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.530574    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-tls-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:09.532868 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.530603    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-home\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:09.558853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.536952    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-dshm" pod "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" (UID: "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:09.630099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.630061    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-kube-api-access-s7qb9" pod "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" (UID: "ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c"). InnerVolumeSpecName "kube-api-access-s7qb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:09.651386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.651308    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7qb9\" (UniqueName: \"kubernetes.io/projected/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-kube-api-access-s7qb9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:09.651386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.651343    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c-dshm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:09.659859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.656489    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv" event={"ID":"ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c","Type":"ContainerDied","Data":"e59a442bf20ce6b4a24cf5ef5fbe89d6c479c5a479033aeba7673482b9a0606f"}
Nov 01 06:07:09.659859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.656567    5510 scope.go:122] "RemoveContainer" containerID="5651d53f927f9dcb74891ab03f2fa3fc20253f4add284a4d7691fd7ccaa8a912"
Nov 01 06:07:09.659859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.656794    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"
Nov 01 06:07:09.754003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.748289    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv" podUID="ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 01 06:07:09.885878 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.885799    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"]
Nov 01 06:07:09.891307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:09.890600    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cda04f/test-model-cda04f-kserve-54646dff59-55ksv"]
Nov 01 06:07:11.236582 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:11.236549    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" path="/var/lib/kubelet/pods/ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c/volumes"
Nov 01 06:07:17.546985 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:07:17.546894    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:07:18.608849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:18.608791    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw" podUID="f22f3138-7e24-4e67-a572-10e9e1d2a8e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:07:19.047643 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:19.047607    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"]
Nov 01 06:07:19.097267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:19.097233    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0c68063-dwppc"]
Nov 01 06:07:19.237098 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:19.237065    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69b2902-f3af-448c-bbcd-47fe3dfac911" path="/var/lib/kubelet/pods/e69b2902-f3af-448c-bbcd-47fe3dfac911/volumes"
Nov 01 06:07:19.777370 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:19.777340    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"]
Nov 01 06:07:19.887434 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:19.887400    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"]
Nov 01 06:07:24.778235 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:24.778178    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" podUID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" containerName="payload-processing" containerID="cri-o://d79eeaa933ac865cccd78714a51c7e367d5f883c84badcf7b566f2c1f1a34a44" gracePeriod=25
Nov 01 06:07:24.888137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:24.888100    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" podUID="11a54968-bcf4-4f36-97a7-86b3c2d15306" containerName="payload-pre-processing" containerID="cri-o://a39e7bea7c6a040b724d61ade48fc7f0991226687599ae699a3f6502af682c89" gracePeriod=25
Nov 01 06:07:25.024154 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.024079    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" podUID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:07:25.024154 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.024093    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" podUID="11a54968-bcf4-4f36-97a7-86b3c2d15306" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:07:25.381015 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.380960    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 06:07:25.519203 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.519172    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-plugins-config-volume\") pod \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\" (UID: \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\") "
Nov 01 06:07:25.519393 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.519239    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-kube-api-access-n428x\" (UniqueName: \"kubernetes.io/projected/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-kube-api-access-n428x\") pod \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\" (UID: \"f22f3138-7e24-4e67-a572-10e9e1d2a8e9\") "
Nov 01 06:07:25.519618 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.519597    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-plugins-config-volume" pod "f22f3138-7e24-4e67-a572-10e9e1d2a8e9" (UID: "f22f3138-7e24-4e67-a572-10e9e1d2a8e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:07:25.520021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.519992    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:25.521482 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.521457    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-kube-api-access-n428x" pod "f22f3138-7e24-4e67-a572-10e9e1d2a8e9" (UID: "f22f3138-7e24-4e67-a572-10e9e1d2a8e9"). InnerVolumeSpecName "kube-api-access-n428x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:25.621381 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.621348    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n428x\" (UniqueName: \"kubernetes.io/projected/f22f3138-7e24-4e67-a572-10e9e1d2a8e9-kube-api-access-n428x\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:25.813041 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.813011    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:07:25.832844 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.832802    5510 generic.go:356] "Generic (PLEG): container finished" podID="f22f3138-7e24-4e67-a572-10e9e1d2a8e9" containerID="848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4" exitCode=0
Nov 01 06:07:25.832980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.832861    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"
Nov 01 06:07:25.832980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.832882    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw" event={"ID":"f22f3138-7e24-4e67-a572-10e9e1d2a8e9","Type":"ContainerDied","Data":"848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4"}
Nov 01 06:07:25.832980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.832941    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw" event={"ID":"f22f3138-7e24-4e67-a572-10e9e1d2a8e9","Type":"ContainerDied","Data":"3404d247fccb3d17332fec7a3a447ec67403173e139376b437ddbe94fc030084"}
Nov 01 06:07:25.832980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.832964    5510 scope.go:122] "RemoveContainer" containerID="848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4"
Nov 01 06:07:25.843713 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.843697    5510 scope.go:122] "RemoveContainer" containerID="848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4"
Nov 01 06:07:25.843969 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:07:25.843941    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4\": container with ID starting with 848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4 not found: ID does not exist" containerID="848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4"
Nov 01 06:07:25.844067 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.843973    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4"} err="failed to get container status \"848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4\": rpc error: code = NotFound desc = could not find container \"848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4\": container with ID starting with 848b8a69d5b287ececa19fa29feb3ac493ee777fd779b551e22e07c207664ef4 not found: ID does not exist"
Nov 01 06:07:25.995943 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.995913    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"]
Nov 01 06:07:25.999297 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:25.999276    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d2807-74d866b599-j8wlw"]
Nov 01 06:07:27.255629 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:27.255594    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22f3138-7e24-4e67-a572-10e9e1d2a8e9" path="/var/lib/kubelet/pods/f22f3138-7e24-4e67-a572-10e9e1d2a8e9/volumes"
Nov 01 06:07:35.025212 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:35.025176    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" podUID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:07:35.025212 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:35.025198    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" podUID="11a54968-bcf4-4f36-97a7-86b3c2d15306" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:07:40.071860 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:40.068648    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"]
Nov 01 06:07:40.127440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:40.127385    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a308883-5snqs"]
Nov 01 06:07:41.251540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:41.251506    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f767baa-f942-4f0d-b3d6-4985b62e41b4" path="/var/lib/kubelet/pods/8f767baa-f942-4f0d-b3d6-4985b62e41b4/volumes"
Nov 01 06:07:45.029531 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:07:45.029486    5510 logging.go:55] [core] [Channel #2071 SubChannel #2072]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.57:50052", ServerName: "10.134.0.57:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.57:50052: connect: connection refused"
Nov 01 06:07:45.122392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.122361    5510 generic.go:356] "Generic (PLEG): container finished" podID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" containerID="d79eeaa933ac865cccd78714a51c7e367d5f883c84badcf7b566f2c1f1a34a44" exitCode=0
Nov 01 06:07:45.122561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.122444    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" event={"ID":"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5","Type":"ContainerDied","Data":"d79eeaa933ac865cccd78714a51c7e367d5f883c84badcf7b566f2c1f1a34a44"}
Nov 01 06:07:45.122561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.122543    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" event={"ID":"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5","Type":"ContainerDied","Data":"56e134a55c9d0bd97ddae1616a00b253d624dcd4242faa9633ec054ec8d02808"}
Nov 01 06:07:45.122671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.122566    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e134a55c9d0bd97ddae1616a00b253d624dcd4242faa9633ec054ec8d02808"
Nov 01 06:07:45.124369 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.124345    5510 generic.go:356] "Generic (PLEG): container finished" podID="11a54968-bcf4-4f36-97a7-86b3c2d15306" containerID="a39e7bea7c6a040b724d61ade48fc7f0991226687599ae699a3f6502af682c89" exitCode=0
Nov 01 06:07:45.124478 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.124377    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" event={"ID":"11a54968-bcf4-4f36-97a7-86b3c2d15306","Type":"ContainerDied","Data":"a39e7bea7c6a040b724d61ade48fc7f0991226687599ae699a3f6502af682c89"}
Nov 01 06:07:45.129781 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.129763    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:07:45.185066 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.185027    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-kube-api-access-n8nzp\" (UniqueName: \"kubernetes.io/projected/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-kube-api-access-n8nzp\") pod \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\" (UID: \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\") "
Nov 01 06:07:45.185272 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.185133    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-plugins-config-volume\") pod \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\" (UID: \"16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5\") "
Nov 01 06:07:45.185564 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.185538    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-plugins-config-volume" pod "16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" (UID: "16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:07:45.185766 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.185749    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:45.187938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.187912    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-kube-api-access-n8nzp" pod "16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" (UID: "16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5"). InnerVolumeSpecName "kube-api-access-n8nzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:45.205081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.205062    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:07:45.286276 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.286248    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a54968-bcf4-4f36-97a7-86b3c2d15306-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a54968-bcf4-4f36-97a7-86b3c2d15306-plugins-config-volume\") pod \"11a54968-bcf4-4f36-97a7-86b3c2d15306\" (UID: \"11a54968-bcf4-4f36-97a7-86b3c2d15306\") "
Nov 01 06:07:45.286567 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.286300    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a54968-bcf4-4f36-97a7-86b3c2d15306-kube-api-access-27p54\" (UniqueName: \"kubernetes.io/projected/11a54968-bcf4-4f36-97a7-86b3c2d15306-kube-api-access-27p54\") pod \"11a54968-bcf4-4f36-97a7-86b3c2d15306\" (UID: \"11a54968-bcf4-4f36-97a7-86b3c2d15306\") "
Nov 01 06:07:45.286729 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.286704    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a54968-bcf4-4f36-97a7-86b3c2d15306-plugins-config-volume" pod "11a54968-bcf4-4f36-97a7-86b3c2d15306" (UID: "11a54968-bcf4-4f36-97a7-86b3c2d15306"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:07:45.286877 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.286821    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8nzp\" (UniqueName: \"kubernetes.io/projected/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5-kube-api-access-n8nzp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:45.286948 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.286882    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a54968-bcf4-4f36-97a7-86b3c2d15306-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:45.288815 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.288795    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a54968-bcf4-4f36-97a7-86b3c2d15306-kube-api-access-27p54" pod "11a54968-bcf4-4f36-97a7-86b3c2d15306" (UID: "11a54968-bcf4-4f36-97a7-86b3c2d15306"). InnerVolumeSpecName "kube-api-access-27p54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:45.387807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:45.387731    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27p54\" (UniqueName: \"kubernetes.io/projected/11a54968-bcf4-4f36-97a7-86b3c2d15306-kube-api-access-27p54\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:46.025962 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.025921    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp" podUID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.58:50052\" within 1s: context deadline exceeded"
Nov 01 06:07:46.026141 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:07:46.025992    5510 logging.go:55] [core] [Channel #2069 SubChannel #2070]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.58:50052", ServerName: "10.134.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.58:50052: operation was canceled"
Nov 01 06:07:46.026141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.026005    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:07:46.030017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.029995    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" podUID="11a54968-bcf4-4f36-97a7-86b3c2d15306" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.57:50052\" within 1s: context deadline exceeded"
Nov 01 06:07:46.030334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.030066    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:07:46.030334 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:07:46.030075    5510 logging.go:55] [core] [Channel #2071 SubChannel #2072]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.57:50052", ServerName: "10.134.0.57:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.57:50052: operation was canceled"
Nov 01 06:07:46.131731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.131663    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"
Nov 01 06:07:46.131731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.131700    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k" event={"ID":"11a54968-bcf4-4f36-97a7-86b3c2d15306","Type":"ContainerDied","Data":"9c7f9623007f0dce78208da3745bf4cb6e17cc8c77da66e202969743dab82b20"}
Nov 01 06:07:46.131954 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.131759    5510 scope.go:122] "RemoveContainer" containerID="a39e7bea7c6a040b724d61ade48fc7f0991226687599ae699a3f6502af682c89"
Nov 01 06:07:46.132022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.131979    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"
Nov 01 06:07:46.332850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.326777    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"]
Nov 01 06:07:46.332850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.328649    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cda04f-cfdcb7479-cgjbp"]
Nov 01 06:07:46.455956 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.455875    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"]
Nov 01 06:07:46.459655 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:46.459631    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cda04f-77d97847cf-vhh5k"]
Nov 01 06:07:47.237248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:47.237207    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a54968-bcf4-4f36-97a7-86b3c2d15306" path="/var/lib/kubelet/pods/11a54968-bcf4-4f36-97a7-86b3c2d15306/volumes"
Nov 01 06:07:47.237878 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:47.237859    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" path="/var/lib/kubelet/pods/16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5/volumes"
Nov 01 06:07:47.545392 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:07:47.545310    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:07:51.046435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:51.046397    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"]
Nov 01 06:07:51.096016 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:51.095984    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ddccc16-vmx5n"]
Nov 01 06:07:51.236993 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:07:51.236964    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143b8d85-16f2-4da8-8f7b-f53367a06498" path="/var/lib/kubelet/pods/143b8d85-16f2-4da8-8f7b-f53367a06498/volumes"
Nov 01 06:08:00.466510 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:00.466475    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:08:03.562813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:03.562784    5510 scope.go:122] "RemoveContainer" containerID="efc6b931dbc265cfc1c9db293d72097d841097fcd3b7cad8d18e6a22986abc8f"
Nov 01 06:08:03.572373 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:03.572351    5510 scope.go:122] "RemoveContainer" containerID="a5a8cbed9095bb1c117d36dc1b5bd494a0b7975fc80958bbf4f56b6245fbf1a3"
Nov 01 06:08:03.581946 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:03.581921    5510 scope.go:122] "RemoveContainer" containerID="5577b3ce8a47986b1fee0a9fe3dd6bfd7be74ab70feae933f7313582d6177129"
Nov 01 06:08:03.590229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:03.590209    5510 scope.go:122] "RemoveContainer" containerID="5c586954a98d18213301a7d6c23012337b9d659e5cabe0ab980aa127239b26c0"
Nov 01 06:08:03.598299 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:03.598273    5510 scope.go:122] "RemoveContainer" containerID="d79eeaa933ac865cccd78714a51c7e367d5f883c84badcf7b566f2c1f1a34a44"
Nov 01 06:08:17.047355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:17.047323    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"]
Nov 01 06:08:17.096515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:17.096486    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a31a638-9ph74"]
Nov 01 06:08:17.237461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:17.237418    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894ad3c4-0e49-41c0-ab22-63dbc8544fa0" path="/var/lib/kubelet/pods/894ad3c4-0e49-41c0-ab22-63dbc8544fa0/volumes"
Nov 01 06:08:17.544864 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:08:17.544790    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:08:33.828330 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:33.828299    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:08:39.034731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:39.034699    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"]
Nov 01 06:08:39.035253 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:39.034930    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy" containerID="cri-o://11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c" gracePeriod=30
Nov 01 06:08:39.084810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:39.084781    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"]
Nov 01 06:08:39.085008 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:39.084986    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy" containerID="cri-o://0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d" gracePeriod=30
Nov 01 06:08:44.395505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.395470    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 06:08:44.461473 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461443    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-podinfo\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.461599 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461512    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-envoy\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.461599 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461548    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6e03b3-0dbf-4e33-ac73-49c364611072-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e03b3-0dbf-4e33-ac73-49c364611072-istiod-ca-cert\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.461686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461610    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-token\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.461724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461685    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-kube-api-access-zg4ht\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-kube-api-access-zg4ht\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.461764 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461749    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-socket\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.461802 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461778    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-data\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.461858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461805    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-certs\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.461980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.461959    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-credential-socket\") pod \"bf6e03b3-0dbf-4e33-ac73-49c364611072\" (UID: \"bf6e03b3-0dbf-4e33-ac73-49c364611072\") "
Nov 01 06:08:44.462628 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.462493    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-socket" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.462628 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.462608    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6e03b3-0dbf-4e33-ac73-49c364611072-istiod-ca-cert" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:44.462940 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.462915    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-data" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.463088 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.463052    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-certs" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.463411 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.463388    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-credential-socket" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.464432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.464414    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 06:08:44.465298 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.465278    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-kube-api-access-zg4ht" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "kube-api-access-zg4ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:44.465423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.465405    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-envoy" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.465469 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.465444    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-token" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:44.465947 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.465816    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-podinfo" pod "bf6e03b3-0dbf-4e33-ac73-49c364611072" (UID: "bf6e03b3-0dbf-4e33-ac73-49c364611072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585216    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-token\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-token\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585275    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-certs\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585363    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-envoy\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585391    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-data\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585429    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-kube-api-access-c4gnk\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-kube-api-access-c4gnk\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585501    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fa3477-782d-4d47-b594-cfa53dc6af05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fa3477-782d-4d47-b594-cfa53dc6af05-istiod-ca-cert\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585588    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-socket\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585633    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-podinfo\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.585705    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-credential-socket\") pod \"69fa3477-782d-4d47-b594-cfa53dc6af05\" (UID: \"69fa3477-782d-4d47-b594-cfa53dc6af05\") "
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586254    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg4ht\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-kube-api-access-zg4ht\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586275    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586290    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586302    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586314    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586329    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586341    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586354    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e03b3-0dbf-4e33-ac73-49c364611072-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586366    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e03b3-0dbf-4e33-ac73-49c364611072-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.586795    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-data" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.587442    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fa3477-782d-4d47-b594-cfa53dc6af05-istiod-ca-cert" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.587577    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-credential-socket" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.587820    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-certs" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.588810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.588067    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-socket" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.596421 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.596341    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-kube-api-access-c4gnk" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "kube-api-access-c4gnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:44.596761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.596738    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-podinfo" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:08:44.618908 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.610629    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-token" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:44.622919 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.620950    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-envoy" pod "69fa3477-782d-4d47-b594-cfa53dc6af05" (UID: "69fa3477-782d-4d47-b594-cfa53dc6af05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.687499 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687463    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.687499 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687491    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.687499 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687499    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.687499 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687507    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.687770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687516    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.687770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687524    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.687770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687533    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fa3477-782d-4d47-b594-cfa53dc6af05-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.687770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687542    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gnk\" (UniqueName: \"kubernetes.io/projected/69fa3477-782d-4d47-b594-cfa53dc6af05-kube-api-access-c4gnk\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.687770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.687551    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fa3477-782d-4d47-b594-cfa53dc6af05-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.760718 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.760689    5510 generic.go:356] "Generic (PLEG): container finished" podID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerID="0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d" exitCode=0
Nov 01 06:08:44.760904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.760741    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"
Nov 01 06:08:44.760904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.760776    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" event={"ID":"69fa3477-782d-4d47-b594-cfa53dc6af05","Type":"ContainerDied","Data":"0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d"}
Nov 01 06:08:44.760904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.760840    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk" event={"ID":"69fa3477-782d-4d47-b594-cfa53dc6af05","Type":"ContainerDied","Data":"9f4c3f806268ddf2012209da63c05cb6579f59b5a4216c655886023ac2c9c72e"}
Nov 01 06:08:44.760904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.760868    5510 scope.go:122] "RemoveContainer" containerID="0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d"
Nov 01 06:08:44.762276 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.762254    5510 generic.go:356] "Generic (PLEG): container finished" podID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerID="11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c" exitCode=0
Nov 01 06:08:44.762377 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.762308    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"
Nov 01 06:08:44.762377 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.762343    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" event={"ID":"bf6e03b3-0dbf-4e33-ac73-49c364611072","Type":"ContainerDied","Data":"11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c"}
Nov 01 06:08:44.762477 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.762392    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr" event={"ID":"bf6e03b3-0dbf-4e33-ac73-49c364611072","Type":"ContainerDied","Data":"99bd721ba5690fabe1b166eeebe17fbbb1e39c2651350718a175548b07898528"}
Nov 01 06:08:44.770914 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.770900    5510 scope.go:122] "RemoveContainer" containerID="0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d"
Nov 01 06:08:44.771263 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:08:44.771149    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d\": container with ID starting with 0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d not found: ID does not exist" containerID="0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d"
Nov 01 06:08:44.771263 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.771180    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d"} err="failed to get container status \"0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d\": rpc error: code = NotFound desc = could not find container \"0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d\": container with ID starting with 0f4106a53a7624d300f113fb45ed6906eb308207527ec9b9b92d8a12a49c2d9d not found: ID does not exist"
Nov 01 06:08:44.771263 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.771205    5510 scope.go:122] "RemoveContainer" containerID="11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c"
Nov 01 06:08:44.780297 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.780279    5510 scope.go:122] "RemoveContainer" containerID="11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c"
Nov 01 06:08:44.780535 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:08:44.780518    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c\": container with ID starting with 11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c not found: ID does not exist" containerID="11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c"
Nov 01 06:08:44.780582 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.780542    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c"} err="failed to get container status \"11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c\": rpc error: code = NotFound desc = could not find container \"11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c\": container with ID starting with 11db9aeb0e204c720dc259cd3797f845eca04612e6062606d86795ebf8b8de6c not found: ID does not exist"
Nov 01 06:08:44.924791 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.924711    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"]
Nov 01 06:08:44.928579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:44.928554    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fj6vk"]
Nov 01 06:08:45.043280 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:45.043243    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"]
Nov 01 06:08:45.050398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:45.050343    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-r6pgr"]
Nov 01 06:08:45.236114 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:45.236084    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" path="/var/lib/kubelet/pods/69fa3477-782d-4d47-b594-cfa53dc6af05/volumes"
Nov 01 06:08:45.236543 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:45.236531    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" path="/var/lib/kubelet/pods/bf6e03b3-0dbf-4e33-ac73-49c364611072/volumes"
Nov 01 06:08:47.544810 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:08:47.544772    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:08:57.853315 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853276    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"]
Nov 01 06:08:57.853891 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853874    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f22f3138-7e24-4e67-a572-10e9e1d2a8e9" containerName="payload-pre-processing"
Nov 01 06:08:57.853891 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853892    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22f3138-7e24-4e67-a572-10e9e1d2a8e9" containerName="payload-pre-processing"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853933    5510 cpu_manager.go:414] "containerMap: removing container" podUID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" containerName="payload-processing"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853939    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" containerName="payload-processing"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853947    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" containerName="main"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853953    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" containerName="main"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853963    5510 cpu_manager.go:414] "containerMap: removing container" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853968    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853979    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.853987    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.854008    5510 cpu_manager.go:414] "containerMap: removing container" podUID="11a54968-bcf4-4f36-97a7-86b3c2d15306" containerName="payload-pre-processing"
Nov 01 06:08:57.854017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.854013    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a54968-bcf4-4f36-97a7-86b3c2d15306" containerName="payload-pre-processing"
Nov 01 06:08:57.854307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.854120    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0e62f5-5531-4e7b-a7a2-4dd7270c3a3c" containerName="main"
Nov 01 06:08:57.854307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.854129    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fa3477-782d-4d47-b594-cfa53dc6af05" containerName="istio-proxy"
Nov 01 06:08:57.854307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.854136    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6e03b3-0dbf-4e33-ac73-49c364611072" containerName="istio-proxy"
Nov 01 06:08:57.854307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.854144    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a54968-bcf4-4f36-97a7-86b3c2d15306" containerName="payload-pre-processing"
Nov 01 06:08:57.854307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.854152    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22f3138-7e24-4e67-a572-10e9e1d2a8e9" containerName="payload-pre-processing"
Nov 01 06:08:57.854307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.854163    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b4924e-d4f1-4e9d-b4ef-94ecdd8284c5" containerName="payload-processing"
Nov 01 06:08:57.857460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.857442    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:57.914479 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.914453    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"]
Nov 01 06:08:57.992217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.992182    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8f45a2-dockercfg-rsccc\""
Nov 01 06:08:57.992484 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.992470    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8f45a2\""
Nov 01 06:08:57.997563 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.997532    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6f9\" (UniqueName: \"kubernetes.io/projected/72f6130c-64f5-4dbc-9553-0cc11c35edd8-kube-api-access-9w6f9\") pod \"payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd\" (UID: \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:58.005917 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:57.997900    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6130c-64f5-4dbc-9553-0cc11c35edd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd\" (UID: \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:58.014944 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.014914    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"]
Nov 01 06:08:58.020135 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.020115    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.072087 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.072054    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"]
Nov 01 06:08:58.109733 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.109641    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3b495-c7e6-41e7-b06d-1014e71d0585-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg\" (UID: \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.109917 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.109783    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6130c-64f5-4dbc-9553-0cc11c35edd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd\" (UID: \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:58.109994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.109921    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwlv\" (UniqueName: \"kubernetes.io/projected/0bd3b495-c7e6-41e7-b06d-1014e71d0585-kube-api-access-2wwlv\") pod \"payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg\" (UID: \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.110126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.110104    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6f9\" (UniqueName: \"kubernetes.io/projected/72f6130c-64f5-4dbc-9553-0cc11c35edd8-kube-api-access-9w6f9\") pod \"payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd\" (UID: \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:58.110442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.110425    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6130c-64f5-4dbc-9553-0cc11c35edd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd\" (UID: \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:58.171198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.171174    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6f9\" (UniqueName: \"kubernetes.io/projected/72f6130c-64f5-4dbc-9553-0cc11c35edd8-kube-api-access-9w6f9\") pod \"payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd\" (UID: \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:58.171929 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.171910    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:58.211461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.211432    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3b495-c7e6-41e7-b06d-1014e71d0585-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg\" (UID: \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.211583 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.211567    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwlv\" (UniqueName: \"kubernetes.io/projected/0bd3b495-c7e6-41e7-b06d-1014e71d0585-kube-api-access-2wwlv\") pod \"payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg\" (UID: \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.212091 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.212073    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3b495-c7e6-41e7-b06d-1014e71d0585-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg\" (UID: \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.272400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.272372    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwlv\" (UniqueName: \"kubernetes.io/projected/0bd3b495-c7e6-41e7-b06d-1014e71d0585-kube-api-access-2wwlv\") pod \"payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg\" (UID: \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.343137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.343102    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.550055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.550032    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"]
Nov 01 06:08:58.551411 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:08:58.551388    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f6130c_64f5_4dbc_9553_0cc11c35edd8.slice/crio-b198d6dd3bd2df5d7defc7c74994a9585298a2d6f6b3a451f7e31d888f711e94 WatchSource:0}: Error finding container b198d6dd3bd2df5d7defc7c74994a9585298a2d6f6b3a451f7e31d888f711e94: Status 404 returned error can't find the container with id b198d6dd3bd2df5d7defc7c74994a9585298a2d6f6b3a451f7e31d888f711e94
Nov 01 06:08:58.717297 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.717271    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"]
Nov 01 06:08:58.719400 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:08:58.719371    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd3b495_c7e6_41e7_b06d_1014e71d0585.slice/crio-cafd7600ddd26c963c57c9c0fb2342a3b0974a70b81b64fa54f2c728d2410fc7 WatchSource:0}: Error finding container cafd7600ddd26c963c57c9c0fb2342a3b0974a70b81b64fa54f2c728d2410fc7: Status 404 returned error can't find the container with id cafd7600ddd26c963c57c9c0fb2342a3b0974a70b81b64fa54f2c728d2410fc7
Nov 01 06:08:58.854234 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.854201    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd" event={"ID":"72f6130c-64f5-4dbc-9553-0cc11c35edd8","Type":"ContainerStarted","Data":"ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6"}
Nov 01 06:08:58.854698 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.854260    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd" event={"ID":"72f6130c-64f5-4dbc-9553-0cc11c35edd8","Type":"ContainerStarted","Data":"b198d6dd3bd2df5d7defc7c74994a9585298a2d6f6b3a451f7e31d888f711e94"}
Nov 01 06:08:58.854698 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.854290    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:08:58.856018 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.855996    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg" event={"ID":"0bd3b495-c7e6-41e7-b06d-1014e71d0585","Type":"ContainerStarted","Data":"48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407"}
Nov 01 06:08:58.856141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.856035    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg" event={"ID":"0bd3b495-c7e6-41e7-b06d-1014e71d0585","Type":"ContainerStarted","Data":"cafd7600ddd26c963c57c9c0fb2342a3b0974a70b81b64fa54f2c728d2410fc7"}
Nov 01 06:08:58.856141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.856063    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:08:58.976893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:58.976783    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd" podStartSLOduration=1.97676941 podStartE2EDuration="1.97676941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:08:58.968569789 +0000 UTC m=+1375.846543199" watchObservedRunningTime="2026-11-01 06:08:58.97676941 +0000 UTC m=+1375.854742872"
Nov 01 06:08:59.137852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:08:59.134708    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg" podStartSLOduration=2.13468971 podStartE2EDuration="2.13468971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:08:59.083169375 +0000 UTC m=+1375.961142786" watchObservedRunningTime="2026-11-01 06:08:59.13468971 +0000 UTC m=+1376.012663122"
Nov 01 06:09:03.663141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:03.663103    5510 scope.go:122] "RemoveContainer" containerID="2c7331f98a8b7e54b8dd21aa4674e20e9eb19c430d969a3f8da701374f571bc6"
Nov 01 06:09:03.674146 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:03.674123    5510 scope.go:122] "RemoveContainer" containerID="a46d3016c0a3a3b59ff0d7f10df3a941685d089921d6752973f36c9208fc97d2"
Nov 01 06:09:03.683127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:03.683106    5510 scope.go:122] "RemoveContainer" containerID="557bba4d40186e0a9f323f9bd3253881ba64dfbb2247bb2d1a40216ff364c0cf"
Nov 01 06:09:09.073269 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.073238    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"]
Nov 01 06:09:09.078706 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.078690    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.160696 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.160667    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.160865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.160728    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.160915 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.160867    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.160979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.160961    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.161016 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.160992    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.161061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.161016    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.161116 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.161096    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.161237 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.161223    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkf2r\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-kube-api-access-bkf2r\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.161274 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.161249    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.262689 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.262659    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf2r\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-kube-api-access-bkf2r\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.262689 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.262694    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.262978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.262786    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.262978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.262917    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.262978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.262961    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263144 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263038    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263144 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263043    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263144 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263107    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263295 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263149    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263295 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263189    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263295 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263272    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263421    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263519 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263502    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.263665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.263646    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.265458 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.265433    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.265618 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.265602    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.283730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.283706    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"]
Nov 01 06:09:09.323265 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.323232    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.323563 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.323513    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf2r\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-kube-api-access-bkf2r\") pod \"maas-default-gateway-openshift-default-855f9574cf-xvdnk\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.392083 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.392053    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:09.823172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.823139    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"]
Nov 01 06:09:09.825897 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:09:09.825860    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0e5fb6_ad11_4bc6_a62b_f9810b72fb90.slice/crio-42a3fa888de285639eaee33d333a58ee0c03ff3a937e39a133bbfcbaa4bcc850 WatchSource:0}: Error finding container 42a3fa888de285639eaee33d333a58ee0c03ff3a937e39a133bbfcbaa4bcc850: Status 404 returned error can't find the container with id 42a3fa888de285639eaee33d333a58ee0c03ff3a937e39a133bbfcbaa4bcc850
Nov 01 06:09:09.828920 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.828885    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:09:09.829021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.828964    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:09:09.829021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.829007    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:09:09.865378 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.865360    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:09:09.865480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.865425    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:09:09.925957 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.925923    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" event={"ID":"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90","Type":"ContainerStarted","Data":"0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925"}
Nov 01 06:09:09.926105 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:09.925981    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" event={"ID":"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90","Type":"ContainerStarted","Data":"42a3fa888de285639eaee33d333a58ee0c03ff3a937e39a133bbfcbaa4bcc850"}
Nov 01 06:09:10.217603 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:10.217546    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podStartSLOduration=1.217525545 podStartE2EDuration="1.217525545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:10.214233779 +0000 UTC m=+1387.092207189" watchObservedRunningTime="2026-11-01 06:09:10.217525545 +0000 UTC m=+1387.095498956"
Nov 01 06:09:10.392630 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:10.392596    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:10.394109 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:10.394083    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:10.394243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:10.394138    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:11.398280 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:11.398241    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:11.399767 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:11.399731    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:12.394189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:12.394152    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:12.394364 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:12.394217    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:13.394314 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:13.394276    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:13.396604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:13.394339    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:13.536640 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:13.536600    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:09:14.394224 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:14.394190    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:14.394535 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:14.394500    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:15.393231 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:15.393192    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:15.393529 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:15.393500    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:16.393857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:16.393222    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:16.393857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:16.393285    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:17.392953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:17.392923    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:17.393121 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:17.392979    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:17.549470 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:09:17.549432    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:09:18.393045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:18.393010    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xvdnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 06:09:18.393227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:18.393079    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 06:09:19.558095 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:19.558059    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:19.558569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:19.558545    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:19.559275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:19.559246    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:09:47.545357 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:09:47.545322    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:09:54.042313 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.042269    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"]
Nov 01 06:09:54.046483 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.046463    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.072988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.072964    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.073121 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.073008    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.073121 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.073036    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.073257 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.073235    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.073317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.073286    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.073473 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.073453    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.073542 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.073526    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwdn\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-kube-api-access-8dwdn\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.073590 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.073574    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.073670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.073647    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965e2e7d-daae-459d-9381-c88fd38dc2e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.158799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.158763    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"]
Nov 01 06:09:54.174691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.174664    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965e2e7d-daae-459d-9381-c88fd38dc2e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.174884 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.174710    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.174884 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.174864    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.174934    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.174992    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175153 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175031    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175153 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175090    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175153 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175099    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175405 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175377    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175494 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175422    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175494 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175426    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwdn\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-kube-api-access-8dwdn\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175494 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175465    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175659 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175506    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965e2e7d-daae-459d-9381-c88fd38dc2e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.175659 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.175523    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.178365 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.178338    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.182031 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.182010    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.235250 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.235217    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.235454 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.235432    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwdn\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-kube-api-access-8dwdn\") pod \"maas-default-gateway-openshift-default-855f9574cf-6gf5t\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.360897 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.360803    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:54.789997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.789971    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"]
Nov 01 06:09:54.791900 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:09:54.791874    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965e2e7d_daae_459d_9381_c88fd38dc2e3.slice/crio-f29e29eed0849daacc94ecc483a3b1ad9fe40dc47c5a22de851b66cfe591d0f2 WatchSource:0}: Error finding container f29e29eed0849daacc94ecc483a3b1ad9fe40dc47c5a22de851b66cfe591d0f2: Status 404 returned error can't find the container with id f29e29eed0849daacc94ecc483a3b1ad9fe40dc47c5a22de851b66cfe591d0f2
Nov 01 06:09:54.794137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.794098    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:09:54.794216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.794184    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:09:54.794254 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:54.794223    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:09:55.210238 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:55.210205    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" event={"ID":"965e2e7d-daae-459d-9381-c88fd38dc2e3","Type":"ContainerStarted","Data":"a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0"}
Nov 01 06:09:55.210592 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:55.210267    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" event={"ID":"965e2e7d-daae-459d-9381-c88fd38dc2e3","Type":"ContainerStarted","Data":"f29e29eed0849daacc94ecc483a3b1ad9fe40dc47c5a22de851b66cfe591d0f2"}
Nov 01 06:09:55.329866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:55.329800    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podStartSLOduration=2.32978657 podStartE2EDuration="2.32978657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:55.328702203 +0000 UTC m=+1432.206675614" watchObservedRunningTime="2026-11-01 06:09:55.32978657 +0000 UTC m=+1432.207760052"
Nov 01 06:09:55.361683 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:55.361650    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:09:55.363314 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:55.363289    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6gf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 06:09:55.363446 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:55.363361    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 06:09:56.361776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:56.361745    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6gf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 06:09:56.362179 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:56.361796    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 06:09:57.364855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:57.364185    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6gf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 06:09:57.364855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:57.364240    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 06:09:57.438906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:57.438873    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:09:58.361149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:58.361109    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6gf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 06:09:58.361343 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:58.361171    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 06:09:59.362016 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:59.361983    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6gf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 06:09:59.362385 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:09:59.362034    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 06:10:00.361089 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:00.361056    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6gf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 06:10:00.361268 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:00.361100    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 06:10:01.361344 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:01.361309    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6gf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 06:10:01.361735 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:01.361371    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 06:10:02.361982 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:02.361941    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6gf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 06:10:02.362377 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:02.362025    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 06:10:03.697994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:03.697953    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:10:03.698522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:03.698318    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:10:03.699166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:03.699143    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:10:03.733332 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:03.733303    5510 scope.go:122] "RemoveContainer" containerID="617c6823488c95d5b50891080bd81543f3a7e9798f7c7c00363e10540d2517e8"
Nov 01 06:10:17.392873 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:17.392762    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:10:17.545739 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:17.545703    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:10:22.067708 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.067660    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"]
Nov 01 06:10:22.073163 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.073140    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:22.116060 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.116028    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a331ec-2379-4453-906a-8cd6aa9996e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg\" (UID: \"67a331ec-2379-4453-906a-8cd6aa9996e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:22.116216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.116140    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9wd\" (UniqueName: \"kubernetes.io/projected/67a331ec-2379-4453-906a-8cd6aa9996e0-kube-api-access-4s9wd\") pod \"payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg\" (UID: \"67a331ec-2379-4453-906a-8cd6aa9996e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:22.125085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.125057    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"]
Nov 01 06:10:22.284851 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.280801    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ec5596-dockercfg-6n568\""
Nov 01 06:10:22.284851 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.280914    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ec5596\""
Nov 01 06:10:22.293852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.291214    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a331ec-2379-4453-906a-8cd6aa9996e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg\" (UID: \"67a331ec-2379-4453-906a-8cd6aa9996e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:22.293852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.291348    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9wd\" (UniqueName: \"kubernetes.io/projected/67a331ec-2379-4453-906a-8cd6aa9996e0-kube-api-access-4s9wd\") pod \"payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg\" (UID: \"67a331ec-2379-4453-906a-8cd6aa9996e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:22.293852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.293440    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a331ec-2379-4453-906a-8cd6aa9996e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg\" (UID: \"67a331ec-2379-4453-906a-8cd6aa9996e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:22.304006 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.301632    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"]
Nov 01 06:10:22.307567 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.307074    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:22.309948 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.309922    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"]
Nov 01 06:10:22.352461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.352428    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9wd\" (UniqueName: \"kubernetes.io/projected/67a331ec-2379-4453-906a-8cd6aa9996e0-kube-api-access-4s9wd\") pod \"payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg\" (UID: \"67a331ec-2379-4453-906a-8cd6aa9996e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:22.388711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.388683    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:22.393208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.393185    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn\" (UID: \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:22.393264 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.393227    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbcm9\" (UniqueName: \"kubernetes.io/projected/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-kube-api-access-tbcm9\") pod \"payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn\" (UID: \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:22.494813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.494776    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn\" (UID: \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:22.495012 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.494839    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcm9\" (UniqueName: \"kubernetes.io/projected/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-kube-api-access-tbcm9\") pod \"payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn\" (UID: \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:22.495518 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.495495    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn\" (UID: \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:22.556131 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.556094    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcm9\" (UniqueName: \"kubernetes.io/projected/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-kube-api-access-tbcm9\") pod \"payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn\" (UID: \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:22.635491 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.635415    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:22.770003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:22.769978    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"]
Nov 01 06:10:22.771201 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:10:22.771172    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a331ec_2379_4453_906a_8cd6aa9996e0.slice/crio-79736aa97b106505e34185cf632387209e7f654ceb0aad1757ae1bad023afec4 WatchSource:0}: Error finding container 79736aa97b106505e34185cf632387209e7f654ceb0aad1757ae1bad023afec4: Status 404 returned error can't find the container with id 79736aa97b106505e34185cf632387209e7f654ceb0aad1757ae1bad023afec4
Nov 01 06:10:23.014432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.014406    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"]
Nov 01 06:10:23.015754 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:10:23.015725    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4108c3c4_065e_4e2a_b9c1_f1d134357b9f.slice/crio-c3c2a365e6600bfeba310d9df4ef2855706ee800a53cd491b4ece4fdeda009ae WatchSource:0}: Error finding container c3c2a365e6600bfeba310d9df4ef2855706ee800a53cd491b4ece4fdeda009ae: Status 404 returned error can't find the container with id c3c2a365e6600bfeba310d9df4ef2855706ee800a53cd491b4ece4fdeda009ae
Nov 01 06:10:23.438100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.438000    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg" event={"ID":"67a331ec-2379-4453-906a-8cd6aa9996e0","Type":"ContainerStarted","Data":"627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07"}
Nov 01 06:10:23.438100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.438067    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:23.438100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.438085    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg" event={"ID":"67a331ec-2379-4453-906a-8cd6aa9996e0","Type":"ContainerStarted","Data":"79736aa97b106505e34185cf632387209e7f654ceb0aad1757ae1bad023afec4"}
Nov 01 06:10:23.439652 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.439627    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn" event={"ID":"4108c3c4-065e-4e2a-b9c1-f1d134357b9f","Type":"ContainerStarted","Data":"8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe"}
Nov 01 06:10:23.439779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.439670    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn" event={"ID":"4108c3c4-065e-4e2a-b9c1-f1d134357b9f","Type":"ContainerStarted","Data":"c3c2a365e6600bfeba310d9df4ef2855706ee800a53cd491b4ece4fdeda009ae"}
Nov 01 06:10:23.439779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.439688    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:23.554120 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.554063    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg" podStartSLOduration=1.554048246 podStartE2EDuration="1.554048246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:23.55232079 +0000 UTC m=+1460.430294201" watchObservedRunningTime="2026-11-01 06:10:23.554048246 +0000 UTC m=+1460.432021655"
Nov 01 06:10:23.689673 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:23.685876    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn" podStartSLOduration=1.685855933 podStartE2EDuration="1.685855933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:23.683301682 +0000 UTC m=+1460.561275115" watchObservedRunningTime="2026-11-01 06:10:23.685855933 +0000 UTC m=+1460.563829344"
Nov 01 06:10:34.342245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.342202    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"]
Nov 01 06:10:34.346844 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.346812    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.412853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.403198    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"agc-cred-94875257eb7d\""
Nov 01 06:10:34.412853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.403329    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"openshift-service-ca.crt\""
Nov 01 06:10:34.412853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.403389    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cfe368\""
Nov 01 06:10:34.464067 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.464032    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"payload-processing-external-model-e2e--6d943d76-dockercfg-sm2d2\""
Nov 01 06:10:34.464372 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.464355    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"routing-overlay\""
Nov 01 06:10:34.480850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.480011    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"kube-root-ca.crt\""
Nov 01 06:10:34.485229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.485205    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:10:34.487045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.487023    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:10:34.512737 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.512686    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.513073 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.513048    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.513188 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.513174    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.513266 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.513253    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjmf\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-kube-api-access-6xjmf\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.535524 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.535475    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"]
Nov 01 06:10:34.614398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.614314    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.614398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.614357    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.614632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.614569    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.614700 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.614665    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjmf\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-kube-api-access-6xjmf\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.615083 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.615061    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.615232 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.615209    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.617115 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.617096    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.676909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.676879    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjmf\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-kube-api-access-6xjmf\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:34.961520 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:34.961490    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:35.358126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:35.356659    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"]
Nov 01 06:10:35.583308 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:35.583277    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" event={"ID":"d8882fa7-baac-487c-80ee-f85d50da40e8","Type":"ContainerStarted","Data":"59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890"}
Nov 01 06:10:35.583476 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:35.583331    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" event={"ID":"d8882fa7-baac-487c-80ee-f85d50da40e8","Type":"ContainerStarted","Data":"a6a7700f1fd757c32c6ede8fbd0bb7dc7b3e58b490cc990c6761e43310e38907"}
Nov 01 06:10:35.583476 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:35.583351    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:35.711744 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:35.711676    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" podStartSLOduration=1.711654923 podStartE2EDuration="1.711654923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:35.711120502 +0000 UTC m=+1472.589093913" watchObservedRunningTime="2026-11-01 06:10:35.711654923 +0000 UTC m=+1472.589628334"
Nov 01 06:10:35.746317 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:35.746262    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-94875257eb7d: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:35.746317 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:35.746302    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:35.746596 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:35.746383    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials podName:d8882fa7-baac-487c-80ee-f85d50da40e8 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:36.246360746 +0000 UTC m=+1473.124334155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8") : secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:36.267480 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:36.267433    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-94875257eb7d: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:36.267480 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:36.267472    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:36.267774 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:36.267617    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials podName:d8882fa7-baac-487c-80ee-f85d50da40e8 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:37.267601022 +0000 UTC m=+1474.145574429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8") : secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:37.278338 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:37.278303    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-94875257eb7d: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:37.278338 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:37.278334    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:37.278793 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:37.278383    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials podName:d8882fa7-baac-487c-80ee-f85d50da40e8 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:39.278365337 +0000 UTC m=+1476.156338748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8") : secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:39.306120 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:39.306085    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-94875257eb7d: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:39.306120 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:39.306112    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:39.306677 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:39.306169    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials podName:d8882fa7-baac-487c-80ee-f85d50da40e8 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:43.306148362 +0000 UTC m=+1480.184121764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8") : secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:39.559707 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:39.559616    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"]
Nov 01 06:10:42.615026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.614986    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"]
Nov 01 06:10:42.619328 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.619311    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.677604    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-envoy\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.677682    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798c5\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-kube-api-access-798c5\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.677756    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istiod-ca-cert\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.677819    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-data\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.677916    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-token\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.677978    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-certs\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.678064    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-credential-socket\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.678085    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-podinfo\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.679863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.678121    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-socket\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.711506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.711472    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7d25db-openshift-default-dockercfg-xs4wt\""
Nov 01 06:10:42.807969 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.807928    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istiod-ca-cert\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.808158 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.808009    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-data\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.808158 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.808076    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-token\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.808158 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.808139    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-certs\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.808334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.808231    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-credential-socket\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.808334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.808260    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-podinfo\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.808334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.808305    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-socket\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.808488 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.808390    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-envoy\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.808488 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.808434    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798c5\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-kube-api-access-798c5\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.831075 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.809684    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istiod-ca-cert\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.845076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.835431    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-certs\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.846006 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.845975    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-credential-socket\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.862798 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.856528    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"]
Nov 01 06:10:42.863115 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.861421    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-socket\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.875761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.875685    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-data\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.920849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.914140    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-token\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.920849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.917380    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-podinfo\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.921432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.921400    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-envoy\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.929768 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.929733    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-798c5\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-kube-api-access-798c5\") pod \"e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") " pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:42.937159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:42.936932    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:43.324332 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:43.324292    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-94875257eb7d: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:43.324332 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:43.324321    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw: secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:43.324602 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:43.324386    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials podName:d8882fa7-baac-487c-80ee-f85d50da40e8 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:51.324369436 +0000 UTC m=+1488.202342840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8") : secret "agc-cred-94875257eb7d" not found
Nov 01 06:10:43.374394 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:43.374365    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"]
Nov 01 06:10:43.375701 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:10:43.375673    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3b79fb_4201_4024_b2d4_fdeb139fef14.slice/crio-5b6664e44018d73d90c43e901a4e7ec08dd7ceb14cea82ccc1a0fa10eeebc171 WatchSource:0}: Error finding container 5b6664e44018d73d90c43e901a4e7ec08dd7ceb14cea82ccc1a0fa10eeebc171: Status 404 returned error can't find the container with id 5b6664e44018d73d90c43e901a4e7ec08dd7ceb14cea82ccc1a0fa10eeebc171
Nov 01 06:10:43.378324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:43.378293    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:10:43.378383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:43.378356    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:10:43.378429 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:43.378387    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:10:43.643558 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:43.643464    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" event={"ID":"9e3b79fb-4201-4024-b2d4-fdeb139fef14","Type":"ContainerStarted","Data":"feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9"}
Nov 01 06:10:43.643558 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:43.643510    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" event={"ID":"9e3b79fb-4201-4024-b2d4-fdeb139fef14","Type":"ContainerStarted","Data":"5b6664e44018d73d90c43e901a4e7ec08dd7ceb14cea82ccc1a0fa10eeebc171"}
Nov 01 06:10:43.766111 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:43.766065    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" podStartSLOduration=1.766047426 podStartE2EDuration="1.766047426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:43.762208762 +0000 UTC m=+1480.640182205" watchObservedRunningTime="2026-11-01 06:10:43.766047426 +0000 UTC m=+1480.644020841"
Nov 01 06:10:43.937977 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:43.937894    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:44.560590 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.560555    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" podUID="d8882fa7-baac-487c-80ee-f85d50da40e8" containerName="payload-processing" containerID="cri-o://59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890" gracePeriod=25
Nov 01 06:10:44.819203 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.819124    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:44.851506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.851476    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials\") pod \"d8882fa7-baac-487c-80ee-f85d50da40e8\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") "
Nov 01 06:10:44.851664 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.851547    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-plugins-config-volume\") pod \"d8882fa7-baac-487c-80ee-f85d50da40e8\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") "
Nov 01 06:10:44.851731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.851715    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-routing-overlay-volume\") pod \"d8882fa7-baac-487c-80ee-f85d50da40e8\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") "
Nov 01 06:10:44.851907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.851884    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-kube-api-access-6xjmf\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-kube-api-access-6xjmf\") pod \"d8882fa7-baac-487c-80ee-f85d50da40e8\" (UID: \"d8882fa7-baac-487c-80ee-f85d50da40e8\") "
Nov 01 06:10:44.852040 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.851964    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-plugins-config-volume" pod "d8882fa7-baac-487c-80ee-f85d50da40e8" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:44.852099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.852077    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-routing-overlay-volume" pod "d8882fa7-baac-487c-80ee-f85d50da40e8" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:44.852427 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.852376    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:44.852427 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.852405    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8882fa7-baac-487c-80ee-f85d50da40e8-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:44.853844 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.853812    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials" pod "d8882fa7-baac-487c-80ee-f85d50da40e8" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:44.854032 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.854016    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-kube-api-access-6xjmf" pod "d8882fa7-baac-487c-80ee-f85d50da40e8" (UID: "d8882fa7-baac-487c-80ee-f85d50da40e8"). InnerVolumeSpecName "kube-api-access-6xjmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:44.943225 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.943196    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:44.954067 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.954040    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjmf\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-kube-api-access-6xjmf\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:44.954067 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:44.954064    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8882fa7-baac-487c-80ee-f85d50da40e8-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:45.657436 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.657354    5510 generic.go:356] "Generic (PLEG): container finished" podID="d8882fa7-baac-487c-80ee-f85d50da40e8" containerID="59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890" exitCode=0
Nov 01 06:10:45.657436 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.657401    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"
Nov 01 06:10:45.657680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.657440    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" event={"ID":"d8882fa7-baac-487c-80ee-f85d50da40e8","Type":"ContainerDied","Data":"59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890"}
Nov 01 06:10:45.657680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.657499    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw" event={"ID":"d8882fa7-baac-487c-80ee-f85d50da40e8","Type":"ContainerDied","Data":"a6a7700f1fd757c32c6ede8fbd0bb7dc7b3e58b490cc990c6761e43310e38907"}
Nov 01 06:10:45.657680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.657529    5510 scope.go:122] "RemoveContainer" containerID="59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890"
Nov 01 06:10:45.657875 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.657772    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:45.659085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.658935    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:10:45.671881 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.671465    5510 scope.go:122] "RemoveContainer" containerID="59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890"
Nov 01 06:10:45.672085 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:45.672059    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890\": container with ID starting with 59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890 not found: ID does not exist" containerID="59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890"
Nov 01 06:10:45.672138 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.672095    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890"} err="failed to get container status \"59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890\": rpc error: code = NotFound desc = could not find container \"59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890\": container with ID starting with 59367f8eda905bc482c16de2794d7fe6ccdda9f9300fc1f608051f9ec1136890 not found: ID does not exist"
Nov 01 06:10:45.947888 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.947747    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"]
Nov 01 06:10:45.950529 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:45.950507    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-6dcbsxnjw"]
Nov 01 06:10:47.237025 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:47.236993    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8882fa7-baac-487c-80ee-f85d50da40e8" path="/var/lib/kubelet/pods/d8882fa7-baac-487c-80ee-f85d50da40e8/volumes"
Nov 01 06:10:47.544301 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:10:47.544217    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:10:51.548548 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.548496    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"]
Nov 01 06:10:51.556350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.556322    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d8882fa7-baac-487c-80ee-f85d50da40e8" containerName="payload-processing"
Nov 01 06:10:51.556553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.556538    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8882fa7-baac-487c-80ee-f85d50da40e8" containerName="payload-processing"
Nov 01 06:10:51.556927 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.556911    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8882fa7-baac-487c-80ee-f85d50da40e8" containerName="payload-processing"
Nov 01 06:10:51.579366 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.579327    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:51.642078 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.641399    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"]
Nov 01 06:10:51.647453 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.645460    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7d25db-dockercfg-gvvrq\""
Nov 01 06:10:51.647453 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.645572    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7d25db\""
Nov 01 06:10:51.667905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.667870    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qfp\" (UniqueName: \"kubernetes.io/projected/59be7ba7-9f54-41d6-a50a-916a7605e821-kube-api-access-l7qfp\") pod \"payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s\" (UID: \"59be7ba7-9f54-41d6-a50a-916a7605e821\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:51.668120 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.668098    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59be7ba7-9f54-41d6-a50a-916a7605e821-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s\" (UID: \"59be7ba7-9f54-41d6-a50a-916a7605e821\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:51.769843 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.769801    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59be7ba7-9f54-41d6-a50a-916a7605e821-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s\" (UID: \"59be7ba7-9f54-41d6-a50a-916a7605e821\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:51.770016 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.769950    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qfp\" (UniqueName: \"kubernetes.io/projected/59be7ba7-9f54-41d6-a50a-916a7605e821-kube-api-access-l7qfp\") pod \"payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s\" (UID: \"59be7ba7-9f54-41d6-a50a-916a7605e821\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:51.770429 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.770409    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59be7ba7-9f54-41d6-a50a-916a7605e821-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s\" (UID: \"59be7ba7-9f54-41d6-a50a-916a7605e821\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:51.830444 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.830379    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qfp\" (UniqueName: \"kubernetes.io/projected/59be7ba7-9f54-41d6-a50a-916a7605e821-kube-api-access-l7qfp\") pod \"payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s\" (UID: \"59be7ba7-9f54-41d6-a50a-916a7605e821\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:51.923977 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:51.923943    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:52.300487 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:52.300458    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"]
Nov 01 06:10:52.301764 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:10:52.301736    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59be7ba7_9f54_41d6_a50a_916a7605e821.slice/crio-a454a426e4c84d0e9faad253fc9ea6e65c38e7bbd14f668c798c8703a3d857e2 WatchSource:0}: Error finding container a454a426e4c84d0e9faad253fc9ea6e65c38e7bbd14f668c798c8703a3d857e2: Status 404 returned error can't find the container with id a454a426e4c84d0e9faad253fc9ea6e65c38e7bbd14f668c798c8703a3d857e2
Nov 01 06:10:52.701230 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:52.701138    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s" event={"ID":"59be7ba7-9f54-41d6-a50a-916a7605e821","Type":"ContainerStarted","Data":"3a9f17eb47b6a886b5def3b27e6e6805cb4983bf4b578d1e18079afceed633c2"}
Nov 01 06:10:52.701230 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:52.701196    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s" event={"ID":"59be7ba7-9f54-41d6-a50a-916a7605e821","Type":"ContainerStarted","Data":"a454a426e4c84d0e9faad253fc9ea6e65c38e7bbd14f668c798c8703a3d857e2"}
Nov 01 06:10:52.701716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:52.701331    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:10:52.818626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:10:52.818372    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s" podStartSLOduration=1.8183532869999999 podStartE2EDuration="1.818353287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:52.816904198 +0000 UTC m=+1489.694877613" watchObservedRunningTime="2026-11-01 06:10:52.818353287 +0000 UTC m=+1489.696326702"
Nov 01 06:11:03.709907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:03.709880    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:11:09.713998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:09.713962    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"]
Nov 01 06:11:09.714778 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:09.714738    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp" podUID="c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" containerName="maas-api" containerID="cri-o://80f16483da39f8804768b9078a6cf9a5b963414db75b96237aaaaeabfea17166" gracePeriod=30
Nov 01 06:11:09.841817 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:09.841784    5510 generic.go:356] "Generic (PLEG): container finished" podID="c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" containerID="80f16483da39f8804768b9078a6cf9a5b963414db75b96237aaaaeabfea17166" exitCode=0
Nov 01 06:11:09.841987 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:09.841893    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp" event={"ID":"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409","Type":"ContainerDied","Data":"80f16483da39f8804768b9078a6cf9a5b963414db75b96237aaaaeabfea17166"}
Nov 01 06:11:10.048788 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.048462    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:11:10.160849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.160790    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-kube-api-access-tmhgc\" (UniqueName: \"kubernetes.io/projected/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-kube-api-access-tmhgc\") pod \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") "
Nov 01 06:11:10.161026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.160864    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-metrics-certs\") pod \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") "
Nov 01 06:11:10.161026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.160943    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-openshift-service-ca\") pod \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") "
Nov 01 06:11:10.161026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.160981    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls\") pod \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\" (UID: \"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409\") "
Nov 01 06:11:10.161300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.161271    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-openshift-service-ca" pod "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" (UID: "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:10.161605 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.161530    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:10.163175 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.163146    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls" pod "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" (UID: "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:10.163273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.163220    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-metrics-certs" pod "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" (UID: "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:10.163273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.163231    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-kube-api-access-tmhgc" pod "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" (UID: "c1dd4aa6-2f7a-4762-b9e9-6dcf10876409"). InnerVolumeSpecName "kube-api-access-tmhgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:11:10.262584 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.262549    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:10.262584 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.262588    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmhgc\" (UniqueName: \"kubernetes.io/projected/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-kube-api-access-tmhgc\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:10.262964 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.262606    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:10.849131 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.849102    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"
Nov 01 06:11:10.849607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.849150    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp" event={"ID":"c1dd4aa6-2f7a-4762-b9e9-6dcf10876409","Type":"ContainerDied","Data":"449a44a93db97769f73b3e52b013944b943daaf272c61e97816b7cfa2c54c565"}
Nov 01 06:11:10.849607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:10.849223    5510 scope.go:122] "RemoveContainer" containerID="80f16483da39f8804768b9078a6cf9a5b963414db75b96237aaaaeabfea17166"
Nov 01 06:11:11.013131 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:11.013087    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"]
Nov 01 06:11:11.062128 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:11.062099    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58d523-56f9676765-2fzgp"]
Nov 01 06:11:11.236615 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:11.236589    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" path="/var/lib/kubelet/pods/c1dd4aa6-2f7a-4762-b9e9-6dcf10876409/volumes"
Nov 01 06:11:14.767776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:14.767741    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:11:17.546280 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:11:17.546244    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:11:35.876545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:35.876511    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:11:37.627806 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.627771    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"]
Nov 01 06:11:37.628423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.628408    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" containerName="maas-api"
Nov 01 06:11:37.628469 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.628425    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" containerName="maas-api"
Nov 01 06:11:37.628537 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.628528    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dd4aa6-2f7a-4762-b9e9-6dcf10876409" containerName="maas-api"
Nov 01 06:11:37.632277 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.632261    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.687609 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.687204    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-7d25db-serving-cert\""
Nov 01 06:11:37.721905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.721867    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928d5d5f-9565-45d6-985a-c0bff8efa61b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.721905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.721904    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.722074 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.721925    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgqb7\" (UniqueName: \"kubernetes.io/projected/928d5d5f-9565-45d6-985a-c0bff8efa61b-kube-api-access-tgqb7\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.722155 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.722123    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.739939 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.739916    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"]
Nov 01 06:11:37.822888 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.822860    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928d5d5f-9565-45d6-985a-c0bff8efa61b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.822888 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.822892    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.823141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.822916    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgqb7\" (UniqueName: \"kubernetes.io/projected/928d5d5f-9565-45d6-985a-c0bff8efa61b-kube-api-access-tgqb7\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.823141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.822977    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.823141 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:11:37.823084    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-serving-cert: secret "maas-api-e2e-extmod-b-7d25db-serving-cert" not found
Nov 01 06:11:37.823141 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:11:37.823142    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls podName:928d5d5f-9565-45d6-985a-c0bff8efa61b nodeName:}" failed. No retries permitted until 2026-11-01 06:11:38.323124527 +0000 UTC m=+1535.201097929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls") pod "maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc" (UID: "928d5d5f-9565-45d6-985a-c0bff8efa61b") : secret "maas-api-e2e-extmod-b-7d25db-serving-cert" not found
Nov 01 06:11:37.823567 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.823550    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928d5d5f-9565-45d6-985a-c0bff8efa61b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.825365 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.825338    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:37.929267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:37.928797    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgqb7\" (UniqueName: \"kubernetes.io/projected/928d5d5f-9565-45d6-985a-c0bff8efa61b-kube-api-access-tgqb7\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:38.341852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:38.338867    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:38.363145 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:38.363112    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:38.546194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:38.546165    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:38.923781 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:38.923756    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"]
Nov 01 06:11:38.925576 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:11:38.925549    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928d5d5f_9565_45d6_985a_c0bff8efa61b.slice/crio-63cbb3d9e9ad9ec5aa5d293b4fbdf92773252620ec3bf4898c0d67122e9437b2 WatchSource:0}: Error finding container 63cbb3d9e9ad9ec5aa5d293b4fbdf92773252620ec3bf4898c0d67122e9437b2: Status 404 returned error can't find the container with id 63cbb3d9e9ad9ec5aa5d293b4fbdf92773252620ec3bf4898c0d67122e9437b2
Nov 01 06:11:38.927041 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:38.927022    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:11:39.018252 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:39.018220    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc" event={"ID":"928d5d5f-9565-45d6-985a-c0bff8efa61b","Type":"ContainerStarted","Data":"63cbb3d9e9ad9ec5aa5d293b4fbdf92773252620ec3bf4898c0d67122e9437b2"}
Nov 01 06:11:40.026286 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:40.026252    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc" event={"ID":"928d5d5f-9565-45d6-985a-c0bff8efa61b","Type":"ContainerStarted","Data":"de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57"}
Nov 01 06:11:40.026779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:40.026318    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:40.144337 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:40.144238    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc" podStartSLOduration=2.248889139 podStartE2EDuration="3.144222609s" totalImagesPullingTime="895.33347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:40.142343564 +0000 UTC m=+1537.020316974" watchObservedRunningTime="2026-11-01 06:11:40.144222609 +0000 UTC m=+1537.022196041"
Nov 01 06:11:46.040675 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:11:46.040603    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:11:47.546118 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:11:47.546071    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:12:01.087589 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:01.086805    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": read tcp 10.134.0.2:54200->10.134.0.94:15021: read: connection reset by peer" start-of-body=
Nov 01 06:12:01.087589 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:01.086903    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": read tcp 10.134.0.2:54200->10.134.0.94:15021: read: connection reset by peer"
Nov 01 06:12:01.162841 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:01.162798    5510 generic.go:356] "Generic (PLEG): container finished" podID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerID="feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9" exitCode=0
Nov 01 06:12:01.162979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:01.162866    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" event={"ID":"9e3b79fb-4201-4024-b2d4-fdeb139fef14","Type":"ContainerDied","Data":"feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9"}
Nov 01 06:12:01.163321 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:01.163307    5510 scope.go:122] "RemoveContainer" containerID="feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9"
Nov 01 06:12:01.164762 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:01.164732    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:12:01.164853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:01.164804    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:12:01.164898 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:01.164871    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:12:02.195368 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:02.195327    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" event={"ID":"9e3b79fb-4201-4024-b2d4-fdeb139fef14","Type":"ContainerStarted","Data":"2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d"}
Nov 01 06:12:02.937886 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:02.937817    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:12:02.938136 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:02.938114    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 06:12:02.938222 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:02.938170    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 06:12:03.937421 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:03.937390    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 06:12:03.937795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:03.937441    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 06:12:04.938468 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:04.938432    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 06:12:04.938879 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:04.938498    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 06:12:05.937722 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:05.937679    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 06:12:05.937944 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:05.937749    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 06:12:06.937602 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:06.937564    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 06:12:06.938136 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:06.937638    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 06:12:07.941522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:07.941491    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:12:07.942005 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:07.941947    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:12:07.942590 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:07.942569    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:12:09.889419 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:09.889390    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:12:11.794615 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:11.794579    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"]
Nov 01 06:12:11.795086 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:11.794806    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb" podUID="4d269181-c1ba-4620-a02b-eb3ac1c68da9" containerName="maas-api" containerID="cri-o://49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778" gracePeriod=30
Nov 01 06:12:12.136673 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.136646    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:12:12.264045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.263934    5510 generic.go:356] "Generic (PLEG): container finished" podID="4d269181-c1ba-4620-a02b-eb3ac1c68da9" containerID="49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778" exitCode=0
Nov 01 06:12:12.264045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.264030    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb" event={"ID":"4d269181-c1ba-4620-a02b-eb3ac1c68da9","Type":"ContainerDied","Data":"49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778"}
Nov 01 06:12:12.264324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.264080    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb" event={"ID":"4d269181-c1ba-4620-a02b-eb3ac1c68da9","Type":"ContainerDied","Data":"0e9ba5b7e22cf2bf4cac69fd19e032b55e194ca14c48c04c1f891d7d7ae0a8ef"}
Nov 01 06:12:12.264324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.264109    5510 scope.go:122] "RemoveContainer" containerID="49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778"
Nov 01 06:12:12.264324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.264270    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"
Nov 01 06:12:12.288737 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.287205    5510 scope.go:122] "RemoveContainer" containerID="49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778"
Nov 01 06:12:12.290468 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:12:12.290339    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778\": container with ID starting with 49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778 not found: ID does not exist" containerID="49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778"
Nov 01 06:12:12.290468 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.290386    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778"} err="failed to get container status \"49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778\": rpc error: code = NotFound desc = could not find container \"49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778\": container with ID starting with 49ee2e47f50cb20ea493254491f2b8921c51981ecb2853731eaa2380e76d7778 not found: ID does not exist"
Nov 01 06:12:12.342809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.342780    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d269181-c1ba-4620-a02b-eb3ac1c68da9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d269181-c1ba-4620-a02b-eb3ac1c68da9-openshift-service-ca\") pod \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") "
Nov 01 06:12:12.342998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.342920    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-metrics-certs\") pod \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") "
Nov 01 06:12:12.342998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.342961    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d269181-c1ba-4620-a02b-eb3ac1c68da9-kube-api-access-mwwkt\" (UniqueName: \"kubernetes.io/projected/4d269181-c1ba-4620-a02b-eb3ac1c68da9-kube-api-access-mwwkt\") pod \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") "
Nov 01 06:12:12.343111 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.343013    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls\") pod \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\" (UID: \"4d269181-c1ba-4620-a02b-eb3ac1c68da9\") "
Nov 01 06:12:12.343882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.343819    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d269181-c1ba-4620-a02b-eb3ac1c68da9-openshift-service-ca" pod "4d269181-c1ba-4620-a02b-eb3ac1c68da9" (UID: "4d269181-c1ba-4620-a02b-eb3ac1c68da9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:12.344064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.344044    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d269181-c1ba-4620-a02b-eb3ac1c68da9-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:12.352275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.352192    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls" pod "4d269181-c1ba-4620-a02b-eb3ac1c68da9" (UID: "4d269181-c1ba-4620-a02b-eb3ac1c68da9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:12:12.352797 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.352761    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-metrics-certs" pod "4d269181-c1ba-4620-a02b-eb3ac1c68da9" (UID: "4d269181-c1ba-4620-a02b-eb3ac1c68da9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:12:12.355422 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.355396    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d269181-c1ba-4620-a02b-eb3ac1c68da9-kube-api-access-mwwkt" pod "4d269181-c1ba-4620-a02b-eb3ac1c68da9" (UID: "4d269181-c1ba-4620-a02b-eb3ac1c68da9"). InnerVolumeSpecName "kube-api-access-mwwkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:12.444754 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.444727    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:12.444754 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.444754    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwwkt\" (UniqueName: \"kubernetes.io/projected/4d269181-c1ba-4620-a02b-eb3ac1c68da9-kube-api-access-mwwkt\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:12.444965 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.444765    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d269181-c1ba-4620-a02b-eb3ac1c68da9-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:12.687455 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.687325    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"]
Nov 01 06:12:12.691741 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:12.691710    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d3f8d-749c4f5956-kl4jb"]
Nov 01 06:12:13.237026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:13.236991    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d269181-c1ba-4620-a02b-eb3ac1c68da9" path="/var/lib/kubelet/pods/4d269181-c1ba-4620-a02b-eb3ac1c68da9/volumes"
Nov 01 06:12:17.165836 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.165804    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"]
Nov 01 06:12:17.166353 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.166057    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy" containerID="cri-o://cf7d13c44235a47a57edc12548f7ebe8d10b0a97de822ed0613e65741c71a375" gracePeriod=30
Nov 01 06:12:17.445260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.445161    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"]
Nov 01 06:12:17.446087 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.446063    5510 cpu_manager.go:414] "containerMap: removing container" podUID="4d269181-c1ba-4620-a02b-eb3ac1c68da9" containerName="maas-api"
Nov 01 06:12:17.446087 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.446086    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d269181-c1ba-4620-a02b-eb3ac1c68da9" containerName="maas-api"
Nov 01 06:12:17.446281 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.446245    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d269181-c1ba-4620-a02b-eb3ac1c68da9" containerName="maas-api"
Nov 01 06:12:17.451421 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.451396    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.497248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.497218    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"]
Nov 01 06:12:17.504058 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.504027    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"kube-root-ca.crt\""
Nov 01 06:12:17.504220 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.504199    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3986bf\""
Nov 01 06:12:17.504300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.504223    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"routing-overlay\""
Nov 01 06:12:17.504300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.504264    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"openshift-service-ca.crt\""
Nov 01 06:12:17.541933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.541895    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-c01d3031634c\""
Nov 01 06:12:17.541933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.541922    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-86a3915c0bf5\""
Nov 01 06:12:17.542128 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.541909    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-d344d6f1c604\""
Nov 01 06:12:17.542128 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.541895    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-3feee30ca48b\""
Nov 01 06:12:17.542128 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.541921    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-e2e--cf3b8510-dockercfg-ftjt9\""
Nov 01 06:12:17.542128 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.541925    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-28460a5a0934\""
Nov 01 06:12:17.546609 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:12:17.546577    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:12:17.558418 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.558392    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.558532 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.558450    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.558689 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.558669    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.558727 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.558719    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92b9\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-kube-api-access-x92b9\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.660458 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.660419    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.660637 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.660463    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92b9\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-kube-api-access-x92b9\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.660637 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.660518    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.660637 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.660553    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.661182 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.661158    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.661281 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.661185    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.663369 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.663343    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.721616 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.721579    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92b9\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-kube-api-access-x92b9\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:17.765155 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:17.765131    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:18.153673 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:18.153641    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"]
Nov 01 06:12:18.154923 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:12:18.154888    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e374f81_bec1_477b_8ee1_e5631744acb0.slice/crio-1b56ed1ebf48febc44349245eafd77bef92029ee7b97f6ee2791538b7505cdf4 WatchSource:0}: Error finding container 1b56ed1ebf48febc44349245eafd77bef92029ee7b97f6ee2791538b7505cdf4: Status 404 returned error can't find the container with id 1b56ed1ebf48febc44349245eafd77bef92029ee7b97f6ee2791538b7505cdf4
Nov 01 06:12:18.378585 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:18.378544    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk" event={"ID":"9e374f81-bec1-477b-8ee1-e5631744acb0","Type":"ContainerStarted","Data":"7fe5d34b0b2d31cb67f7ea2b57af92dbf640e42d49affd489986cc6a2f0f550d"}
Nov 01 06:12:18.379027 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:18.378597    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk" event={"ID":"9e374f81-bec1-477b-8ee1-e5631744acb0","Type":"ContainerStarted","Data":"1b56ed1ebf48febc44349245eafd77bef92029ee7b97f6ee2791538b7505cdf4"}
Nov 01 06:12:18.379027 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:18.378646    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:18.496882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:18.496794    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk" podStartSLOduration=1.496781406 podStartE2EDuration="1.496781406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:12:18.494534934 +0000 UTC m=+1575.372508345" watchObservedRunningTime="2026-11-01 06:12:18.496781406 +0000 UTC m=+1575.374754829"
Nov 01 06:12:22.413238 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.413203    5510 generic.go:356] "Generic (PLEG): container finished" podID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerID="cf7d13c44235a47a57edc12548f7ebe8d10b0a97de822ed0613e65741c71a375" exitCode=0
Nov 01 06:12:22.413683 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.413243    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" event={"ID":"c7af0213-d788-40b0-8d4d-e68f0363f3d7","Type":"ContainerDied","Data":"cf7d13c44235a47a57edc12548f7ebe8d10b0a97de822ed0613e65741c71a375"}
Nov 01 06:12:22.413683 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.413293    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2" event={"ID":"c7af0213-d788-40b0-8d4d-e68f0363f3d7","Type":"ContainerDied","Data":"694cf75f11717cc75cc3ff0e1a4183fc77c0da2aaf89609dabb1848b52da7ea0"}
Nov 01 06:12:22.413683 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.413315    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="694cf75f11717cc75cc3ff0e1a4183fc77c0da2aaf89609dabb1848b52da7ea0"
Nov 01 06:12:22.425081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.425062    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:12:22.520511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520484    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-podinfo\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.520636 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520609    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-envoy\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.520694 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520648    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-kube-api-access-kqq64\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-kube-api-access-kqq64\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.520694 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520667    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-socket\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.520781 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520700    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-token\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-token\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.520864 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520805    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-certs\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.520933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520871    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-data\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.520994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520939    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istiod-ca-cert\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.520994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520968    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-socket" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.520994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.520983    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-credential-socket\") pod \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\" (UID: \"c7af0213-d788-40b0-8d4d-e68f0363f3d7\") "
Nov 01 06:12:22.521205 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.521157    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-data" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.521433 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.521331    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-certs" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.521433 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.521388    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istiod-ca-cert" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:22.521567 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.521516    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-credential-socket" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.521626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.521614    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.521682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.521635    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.521682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.521651    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.521682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.521664    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.523076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.523052    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-token" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:22.523190 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.523169    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-kube-api-access-kqq64" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "kube-api-access-kqq64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:22.523334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.523310    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-podinfo" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:12:22.523334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.523321    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-envoy" pod "c7af0213-d788-40b0-8d4d-e68f0363f3d7" (UID: "c7af0213-d788-40b0-8d4d-e68f0363f3d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.623435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.623400    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.623435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.623430    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqq64\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-kube-api-access-kqq64\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.623435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.623442    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.623671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.623451    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7af0213-d788-40b0-8d4d-e68f0363f3d7-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.623671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:22.623460    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7af0213-d788-40b0-8d4d-e68f0363f3d7-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:23.419927 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:23.419901    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"
Nov 01 06:12:23.634566 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:23.634476    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"]
Nov 01 06:12:23.659751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:23.659724    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-zszz2"]
Nov 01 06:12:25.236477 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:25.236443    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" path="/var/lib/kubelet/pods/c7af0213-d788-40b0-8d4d-e68f0363f3d7/volumes"
Nov 01 06:12:29.387806 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:29.387780    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:12:42.588681 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:42.588647    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:12:47.545087 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:12:47.545051    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:12:56.169510 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.169476    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"]
Nov 01 06:12:56.170107 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.170092    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy"
Nov 01 06:12:56.170161 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.170108    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy"
Nov 01 06:12:56.170244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.170235    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7af0213-d788-40b0-8d4d-e68f0363f3d7" containerName="istio-proxy"
Nov 01 06:12:56.173836 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.173808    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.228445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.228399    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-bdcae79313a2\""
Nov 01 06:12:56.242410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.242376    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.242575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.242491    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbch\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-kube-api-access-rjbch\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.242575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.242554    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.242698 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.242622    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.277938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.277899    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"routing-overlay\""
Nov 01 06:12:56.277938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.277917    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"openshift-service-ca.crt\""
Nov 01 06:12:56.277938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.277901    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7d25db\""
Nov 01 06:12:56.278133 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.277953    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-e2e--a835acb2-dockercfg-kqgqq\""
Nov 01 06:12:56.278133 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.277963    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"kube-root-ca.crt\""
Nov 01 06:12:56.280736 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.280716    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"]
Nov 01 06:12:56.343696 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.343669    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.343814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.343735    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.343906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.343882    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.343973 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.343958    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbch\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-kube-api-access-rjbch\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.344458 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.344435    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.344926 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.344904    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.346386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.346360    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.404674 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.404646    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbch\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-kube-api-access-rjbch\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.488118 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.488092    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:56.866261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:56.866232    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"]
Nov 01 06:12:56.867441 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:12:56.867413    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b38c4e1_a37f_4e62_ac69_f998f082e937.slice/crio-663dc5e222965e865d5b3d571eb919891aa7233c628b27c4112f5be2dcd17c51 WatchSource:0}: Error finding container 663dc5e222965e865d5b3d571eb919891aa7233c628b27c4112f5be2dcd17c51: Status 404 returned error can't find the container with id 663dc5e222965e865d5b3d571eb919891aa7233c628b27c4112f5be2dcd17c51
Nov 01 06:12:57.684858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:57.678642    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d" event={"ID":"2b38c4e1-a37f-4e62-ac69-f998f082e937","Type":"ContainerStarted","Data":"67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe"}
Nov 01 06:12:57.684858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:57.678715    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d" event={"ID":"2b38c4e1-a37f-4e62-ac69-f998f082e937","Type":"ContainerStarted","Data":"663dc5e222965e865d5b3d571eb919891aa7233c628b27c4112f5be2dcd17c51"}
Nov 01 06:12:57.684858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:57.680018    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:12:57.797402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:57.797341    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d" podStartSLOduration=1.797323965 podStartE2EDuration="1.797323965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:12:57.795488775 +0000 UTC m=+1614.673462184" watchObservedRunningTime="2026-11-01 06:12:57.797323965 +0000 UTC m=+1614.675297375"
Nov 01 06:12:58.725650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.725614    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"]
Nov 01 06:12:58.732286 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.732257    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.784612 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.784572    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"]
Nov 01 06:12:58.798719 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.798686    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.798906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.798808    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.798994 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.798935    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788x6\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-kube-api-access-788x6\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.799308 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.799285    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.830531 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.830499    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8f45a2\""
Nov 01 06:12:58.830531 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.830503    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-d6d3484085b2\""
Nov 01 06:12:58.830667 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.830542    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-e2e--cb1d2723-dockercfg-smk8q\""
Nov 01 06:12:58.833515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.833503    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"routing-overlay\""
Nov 01 06:12:58.833598 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.833584    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"kube-root-ca.crt\""
Nov 01 06:12:58.833634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.833591    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"openshift-service-ca.crt\""
Nov 01 06:12:58.900305 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.900276    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.900429 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.900313    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.900429 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.900384    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.900509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.900460    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788x6\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-kube-api-access-788x6\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.901087 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.901045    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.901087 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.901082    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.902962 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.902942    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:58.964402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:58.964381    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-788x6\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-kube-api-access-788x6\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:59.047566 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:59.047496    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:59.427269 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:59.427235    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"]
Nov 01 06:12:59.428349 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:12:59.428319    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a0dd79_6e25_42f3_917b_b19464471a50.slice/crio-3f8154e17f607fe12db7f88edc1b2bd364b7551f4be6414d7cf47429ef953315 WatchSource:0}: Error finding container 3f8154e17f607fe12db7f88edc1b2bd364b7551f4be6414d7cf47429ef953315: Status 404 returned error can't find the container with id 3f8154e17f607fe12db7f88edc1b2bd364b7551f4be6414d7cf47429ef953315
Nov 01 06:12:59.691668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:59.691569    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl" event={"ID":"83a0dd79-6e25-42f3-917b-b19464471a50","Type":"ContainerStarted","Data":"beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb"}
Nov 01 06:12:59.691668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:59.691631    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl" event={"ID":"83a0dd79-6e25-42f3-917b-b19464471a50","Type":"ContainerStarted","Data":"3f8154e17f607fe12db7f88edc1b2bd364b7551f4be6414d7cf47429ef953315"}
Nov 01 06:12:59.691668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:59.691663    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:12:59.808349 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:12:59.808299    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl" podStartSLOduration=1.808284731 podStartE2EDuration="1.808284731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:12:59.807498848 +0000 UTC m=+1616.685472258" watchObservedRunningTime="2026-11-01 06:12:59.808284731 +0000 UTC m=+1616.686258140"
Nov 01 06:13:03.812050 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:03.812021    5510 scope.go:122] "RemoveContainer" containerID="cf7d13c44235a47a57edc12548f7ebe8d10b0a97de822ed0613e65741c71a375"
Nov 01 06:13:06.787788 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:06.787755    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:13:09.695192 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:09.695161    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:13:10.701713 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:10.701682    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:13:17.545718 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:13:17.545630    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:13:34.282434 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:34.282400    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:13:39.053980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:39.053941    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"]
Nov 01 06:13:39.054359 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:39.054173    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy" containerID="cri-o://88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9" gracePeriod=30
Nov 01 06:13:44.317924 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.317900    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:13:44.425597 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425517    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-credential-socket\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.425597 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425558    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-kube-api-access-bmdcx\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-kube-api-access-bmdcx\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.425597 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425590    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-podinfo\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.425905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425663    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-token\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-token\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.425905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425713    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-certs\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.425905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425747    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-envoy\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.425905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425807    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-data\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.425905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425869    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-credential-socket" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:13:44.425905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425877    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-socket\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.426201 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.425939    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istiod-ca-cert\") pod \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\" (UID: \"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5\") "
Nov 01 06:13:44.426201 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426020    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-certs" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:13:44.426201 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426140    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-socket" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:13:44.426333 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426273    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-data" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:13:44.426333 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426315    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istiod-ca-cert" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:13:44.426628 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426603    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:44.426702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426627    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:44.426702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426639    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:44.426702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426649    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:44.426702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.426657    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:44.428653 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.428629    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-kube-api-access-bmdcx" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "kube-api-access-bmdcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:13:44.428760 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.428650    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-token" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:13:44.428760 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.428667    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-podinfo" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:13:44.428760 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.428681    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-envoy" pod "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" (UID: "79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:13:44.527799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.527760    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmdcx\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-kube-api-access-bmdcx\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:44.527799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.527789    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:44.527799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.527799    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:44.527799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:44.527808    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:13:45.020290 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.020253    5510 generic.go:356] "Generic (PLEG): container finished" podID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerID="88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9" exitCode=0
Nov 01 06:13:45.020480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.020315    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"
Nov 01 06:13:45.020480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.020318    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" event={"ID":"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5","Type":"ContainerDied","Data":"88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9"}
Nov 01 06:13:45.020480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.020383    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv" event={"ID":"79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5","Type":"ContainerDied","Data":"c0da1fedda8cbd7276a4c7ba167d8723c24e9af227b597c3a194c282fc4254f1"}
Nov 01 06:13:45.020480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.020407    5510 scope.go:122] "RemoveContainer" containerID="88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9"
Nov 01 06:13:45.031681 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.031662    5510 scope.go:122] "RemoveContainer" containerID="88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9"
Nov 01 06:13:45.032030 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:13:45.032006    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9\": container with ID starting with 88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9 not found: ID does not exist" containerID="88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9"
Nov 01 06:13:45.032136 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.032044    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9"} err="failed to get container status \"88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9\": rpc error: code = NotFound desc = could not find container \"88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9\": container with ID starting with 88d748b0897754c33cf08b6993263ec2bf0e8ed1a70f8928187488074c12fdb9 not found: ID does not exist"
Nov 01 06:13:45.245288 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.245260    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"]
Nov 01 06:13:45.284584 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:45.284513    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-5rmlv"]
Nov 01 06:13:47.236879 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:13:47.236844    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" path="/var/lib/kubelet/pods/79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5/volumes"
Nov 01 06:13:47.545358 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:13:47.545271    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:14:07.301267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:07.301237    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:14:08.321107 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:08.321068    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"]
Nov 01 06:14:08.321515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:08.321272    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy" containerID="cri-o://521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446" gracePeriod=30
Nov 01 06:14:12.546175 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:12.546141    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:14:12.546552 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:12.546209    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:14:13.601691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.601621    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:14:13.710480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710455    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-envoy\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.710480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710485    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-istio-token\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-istio-token\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.710682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710542    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-data\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.710682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710586    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-kube-api-access-hlhvp\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-kube-api-access-hlhvp\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.710682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710627    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-certs\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.710682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710667    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d49d1385-d230-4512-b014-6add6afeb772-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d1385-d230-4512-b014-6add6afeb772-istiod-ca-cert\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.710895 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710707    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-credential-socket\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.710895 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710814    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d49d1385-d230-4512-b014-6add6afeb772-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d1385-d230-4512-b014-6add6afeb772-istio-podinfo\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.711003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710893    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-socket\") pod \"d49d1385-d230-4512-b014-6add6afeb772\" (UID: \"d49d1385-d230-4512-b014-6add6afeb772\") "
Nov 01 06:14:13.711003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710919    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-data" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:14:13.711003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.710974    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-certs" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:14:13.711166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.711055    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-credential-socket" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:14:13.711166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.711121    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49d1385-d230-4512-b014-6add6afeb772-istiod-ca-cert" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:14:13.711325 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.711298    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-socket" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:14:13.711750 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.711729    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:13.711807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.711758    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:13.711807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.711773    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:13.711807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.711787    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d1385-d230-4512-b014-6add6afeb772-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:13.711807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.711801    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:13.713221 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.713194    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-istio-token" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:14:13.713221 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.713208    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d49d1385-d230-4512-b014-6add6afeb772-istio-podinfo" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:14:13.713448 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.713428    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-envoy" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:14:13.713515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.713463    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-kube-api-access-hlhvp" pod "d49d1385-d230-4512-b014-6add6afeb772" (UID: "d49d1385-d230-4512-b014-6add6afeb772"). InnerVolumeSpecName "kube-api-access-hlhvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:14:13.813348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.813320    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d1385-d230-4512-b014-6add6afeb772-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:13.813348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.813343    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:13.813348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.813354    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhvp\" (UniqueName: \"kubernetes.io/projected/d49d1385-d230-4512-b014-6add6afeb772-kube-api-access-hlhvp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:13.813567 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:13.813363    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d1385-d230-4512-b014-6add6afeb772-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:14.216519 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.216472    5510 generic.go:356] "Generic (PLEG): container finished" podID="d49d1385-d230-4512-b014-6add6afeb772" containerID="521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446" exitCode=0
Nov 01 06:14:14.216724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.216590    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"
Nov 01 06:14:14.216724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.216623    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" event={"ID":"d49d1385-d230-4512-b014-6add6afeb772","Type":"ContainerDied","Data":"521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446"}
Nov 01 06:14:14.216724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.216679    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p" event={"ID":"d49d1385-d230-4512-b014-6add6afeb772","Type":"ContainerDied","Data":"7de5dd86a8dd229fd81b7d07019387a60f82c223fdbf6afc23a1e76d80aa87b4"}
Nov 01 06:14:14.216724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.216710    5510 scope.go:122] "RemoveContainer" containerID="521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446"
Nov 01 06:14:14.227531 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.227512    5510 scope.go:122] "RemoveContainer" containerID="521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446"
Nov 01 06:14:14.227786 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:14:14.227768    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446\": container with ID starting with 521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446 not found: ID does not exist" containerID="521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446"
Nov 01 06:14:14.227875 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.227797    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446"} err="failed to get container status \"521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446\": rpc error: code = NotFound desc = could not find container \"521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446\": container with ID starting with 521ac043a161ab42372831d178d7f75a8155ea973c9c1e0742573f7f8b36d446 not found: ID does not exist"
Nov 01 06:14:14.381801 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.381767    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"]
Nov 01 06:14:14.386323 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:14.386296    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dr62p"]
Nov 01 06:14:15.236409 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:15.236371    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d49d1385-d230-4512-b014-6add6afeb772" path="/var/lib/kubelet/pods/d49d1385-d230-4512-b014-6add6afeb772/volumes"
Nov 01 06:14:17.545546 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:14:17.545516    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:14:32.919711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:14:32.919679    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:14:47.545532 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:14:47.545450    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:15:00.179356 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.179323    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"]
Nov 01 06:15:00.180109 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.180082    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy"
Nov 01 06:15:00.180109 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.180106    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy"
Nov 01 06:15:00.180318 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.180170    5510 cpu_manager.go:414] "containerMap: removing container" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy"
Nov 01 06:15:00.180318 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.180179    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy"
Nov 01 06:15:00.180420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.180347    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49d1385-d230-4512-b014-6add6afeb772" containerName="istio-proxy"
Nov 01 06:15:00.180420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.180371    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e2af86-bd15-4cd1-a1d1-3ec94b18c8c5" containerName="istio-proxy"
Nov 01 06:15:00.183816 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.183795    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:00.198139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.198114    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"]
Nov 01 06:15:00.202076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.202057    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"]
Nov 01 06:15:00.202191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.202166    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:00.205807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.205788    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:00.245047 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.245022    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:15:00.279898 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.279864    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7t9\" (UniqueName: \"kubernetes.io/projected/e4cb781f-68e6-4468-86d7-df65be902d09-kube-api-access-8t7t9\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs\" (UID: \"e4cb781f-68e6-4468-86d7-df65be902d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:00.280079 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.279914    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2213f14-67e4-40ad-967c-2b7979b093fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56\" (UID: \"d2213f14-67e4-40ad-967c-2b7979b093fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:00.280079 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.279999    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb781f-68e6-4468-86d7-df65be902d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs\" (UID: \"e4cb781f-68e6-4468-86d7-df65be902d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:00.280199 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.280157    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6fw\" (UniqueName: \"kubernetes.io/projected/d2213f14-67e4-40ad-967c-2b7979b093fc-kube-api-access-mx6fw\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56\" (UID: \"d2213f14-67e4-40ad-967c-2b7979b093fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:00.280248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.280209    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccc4346-1335-4c04-a90b-266c8aaaddf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9\" (UID: \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:00.280488 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.280466    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vhsj\" (UniqueName: \"kubernetes.io/projected/8ccc4346-1335-4c04-a90b-266c8aaaddf1-kube-api-access-9vhsj\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9\" (UID: \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:00.287686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.287659    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"]
Nov 01 06:15:00.335142 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.335077    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"]
Nov 01 06:15:00.339228 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.339197    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"]
Nov 01 06:15:00.381894 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.381858    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7t9\" (UniqueName: \"kubernetes.io/projected/e4cb781f-68e6-4468-86d7-df65be902d09-kube-api-access-8t7t9\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs\" (UID: \"e4cb781f-68e6-4468-86d7-df65be902d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:00.381894 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.381898    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2213f14-67e4-40ad-967c-2b7979b093fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56\" (UID: \"d2213f14-67e4-40ad-967c-2b7979b093fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:00.382143 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.381970    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb781f-68e6-4468-86d7-df65be902d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs\" (UID: \"e4cb781f-68e6-4468-86d7-df65be902d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:00.382143 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.382108    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6fw\" (UniqueName: \"kubernetes.io/projected/d2213f14-67e4-40ad-967c-2b7979b093fc-kube-api-access-mx6fw\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56\" (UID: \"d2213f14-67e4-40ad-967c-2b7979b093fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:00.382259 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.382157    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccc4346-1335-4c04-a90b-266c8aaaddf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9\" (UID: \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:00.382432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.382402    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vhsj\" (UniqueName: \"kubernetes.io/projected/8ccc4346-1335-4c04-a90b-266c8aaaddf1-kube-api-access-9vhsj\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9\" (UID: \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:00.382645 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.382628    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2213f14-67e4-40ad-967c-2b7979b093fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56\" (UID: \"d2213f14-67e4-40ad-967c-2b7979b093fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:00.382703 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.382686    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb781f-68e6-4468-86d7-df65be902d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs\" (UID: \"e4cb781f-68e6-4468-86d7-df65be902d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:00.382967 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.382950    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccc4346-1335-4c04-a90b-266c8aaaddf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9\" (UID: \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:00.443446 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.443381    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7t9\" (UniqueName: \"kubernetes.io/projected/e4cb781f-68e6-4468-86d7-df65be902d09-kube-api-access-8t7t9\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs\" (UID: \"e4cb781f-68e6-4468-86d7-df65be902d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:00.443446 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.443389    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vhsj\" (UniqueName: \"kubernetes.io/projected/8ccc4346-1335-4c04-a90b-266c8aaaddf1-kube-api-access-9vhsj\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9\" (UID: \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:00.443630 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.443521    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6fw\" (UniqueName: \"kubernetes.io/projected/d2213f14-67e4-40ad-967c-2b7979b093fc-kube-api-access-mx6fw\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56\" (UID: \"d2213f14-67e4-40ad-967c-2b7979b093fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:00.498548 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.498516    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:00.518693 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.518669    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:00.525267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.525184    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:00.891888 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.891862    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"]
Nov 01 06:15:00.892214 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:15:00.892192    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccc4346_1335_4c04_a90b_266c8aaaddf1.slice/crio-46b09f11c078db5b2e4405982e943030d51639925674ce077860ef68b8a954f7 WatchSource:0}: Error finding container 46b09f11c078db5b2e4405982e943030d51639925674ce077860ef68b8a954f7: Status 404 returned error can't find the container with id 46b09f11c078db5b2e4405982e943030d51639925674ce077860ef68b8a954f7
Nov 01 06:15:00.926575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.926548    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"]
Nov 01 06:15:00.932014 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.931992    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:00.942076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.942037    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"]
Nov 01 06:15:00.943427 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:15:00.943404    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cb781f_68e6_4468_86d7_df65be902d09.slice/crio-32bdcc823eb340f85a814526fd96e7d5193f395874faef81b542c9a616d16cbe WatchSource:0}: Error finding container 32bdcc823eb340f85a814526fd96e7d5193f395874faef81b542c9a616d16cbe: Status 404 returned error can't find the container with id 32bdcc823eb340f85a814526fd96e7d5193f395874faef81b542c9a616d16cbe
Nov 01 06:15:00.944769 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:15:00.944745    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2213f14_67e4_40ad_967c_2b7979b093fc.slice/crio-008ecaa44b581823a5524d51332a234ea1923c60dbaea661fee2aefc0d19f08a WatchSource:0}: Error finding container 008ecaa44b581823a5524d51332a234ea1923c60dbaea661fee2aefc0d19f08a: Status 404 returned error can't find the container with id 008ecaa44b581823a5524d51332a234ea1923c60dbaea661fee2aefc0d19f08a
Nov 01 06:15:00.945929 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.945911    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"]
Nov 01 06:15:00.986847 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.986810    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:15:00.990700 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.990674    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"]
Nov 01 06:15:00.996701 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.996675    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:00.996839 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.996763    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qcck\" (UniqueName: \"kubernetes.io/projected/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-kube-api-access-6qcck\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:00.996914 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:00.996892    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:01.098496 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.098461    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:01.098640 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.098538    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qcck\" (UniqueName: \"kubernetes.io/projected/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-kube-api-access-6qcck\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:01.098640 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.098580    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:01.099248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.099188    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:01.170480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.170415    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:01.170806 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.170783    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qcck\" (UniqueName: \"kubernetes.io/projected/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-kube-api-access-6qcck\") pod \"maas-api-revoke-sub-31a01dd9-qkcrg\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:01.248998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.248974    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:01.539996 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.539961    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" event={"ID":"d2213f14-67e4-40ad-967c-2b7979b093fc","Type":"ContainerStarted","Data":"527c731d19aaee8fd6e2a08b26d9e7e35fafd33206fae46d85c7106311a7b192"}
Nov 01 06:15:01.540223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.540027    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" event={"ID":"d2213f14-67e4-40ad-967c-2b7979b093fc","Type":"ContainerStarted","Data":"008ecaa44b581823a5524d51332a234ea1923c60dbaea661fee2aefc0d19f08a"}
Nov 01 06:15:01.541635 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.541605    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" event={"ID":"e4cb781f-68e6-4468-86d7-df65be902d09","Type":"ContainerStarted","Data":"b71fc3cb28f8548476b436961f58f21fa577ca3c3a3f6d47c8cc09701717b829"}
Nov 01 06:15:01.541783 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.541650    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" event={"ID":"e4cb781f-68e6-4468-86d7-df65be902d09","Type":"ContainerStarted","Data":"32bdcc823eb340f85a814526fd96e7d5193f395874faef81b542c9a616d16cbe"}
Nov 01 06:15:01.543294 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.543273    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" event={"ID":"8ccc4346-1335-4c04-a90b-266c8aaaddf1","Type":"ContainerStarted","Data":"c06c2fb6113894fd4f3f7555c5ac95218d1b329ed6cd0a5ba4e233f1d09c7be0"}
Nov 01 06:15:01.543396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.543309    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" event={"ID":"8ccc4346-1335-4c04-a90b-266c8aaaddf1","Type":"ContainerStarted","Data":"46b09f11c078db5b2e4405982e943030d51639925674ce077860ef68b8a954f7"}
Nov 01 06:15:01.655240 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.655171    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" podStartSLOduration=1.655154646 podStartE2EDuration="1.655154646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.654058232 +0000 UTC m=+1738.532031645" watchObservedRunningTime="2026-11-01 06:15:01.655154646 +0000 UTC m=+1738.533128057"
Nov 01 06:15:01.701539 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.701504    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"]
Nov 01 06:15:01.702283 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:15:01.702253    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8307b4c_2e1c_4b23_b63c_5bab1266ae39.slice/crio-6e86f9fee59cf688eda6594e26cc80988ebad775f826224ee2457c70aef53d94 WatchSource:0}: Error finding container 6e86f9fee59cf688eda6594e26cc80988ebad775f826224ee2457c70aef53d94: Status 404 returned error can't find the container with id 6e86f9fee59cf688eda6594e26cc80988ebad775f826224ee2457c70aef53d94
Nov 01 06:15:01.769872 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.769788    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" podStartSLOduration=1.769773787 podStartE2EDuration="1.769773787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.768628316 +0000 UTC m=+1738.646601727" watchObservedRunningTime="2026-11-01 06:15:01.769773787 +0000 UTC m=+1738.647747197"
Nov 01 06:15:01.885155 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:01.885093    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" podStartSLOduration=1.88507175 podStartE2EDuration="1.88507175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.882546814 +0000 UTC m=+1738.760520236" watchObservedRunningTime="2026-11-01 06:15:01.88507175 +0000 UTC m=+1738.763045160"
Nov 01 06:15:02.569047 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.569016    5510 generic.go:356] "Generic (PLEG): container finished" podID="e4cb781f-68e6-4468-86d7-df65be902d09" containerID="b71fc3cb28f8548476b436961f58f21fa577ca3c3a3f6d47c8cc09701717b829" exitCode=60
Nov 01 06:15:02.569545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.569212    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" event={"ID":"e4cb781f-68e6-4468-86d7-df65be902d09","Type":"ContainerDied","Data":"b71fc3cb28f8548476b436961f58f21fa577ca3c3a3f6d47c8cc09701717b829"}
Nov 01 06:15:02.569621 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.569606    5510 scope.go:122] "RemoveContainer" containerID="b71fc3cb28f8548476b436961f58f21fa577ca3c3a3f6d47c8cc09701717b829"
Nov 01 06:15:02.573168 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.572819    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg" event={"ID":"c8307b4c-2e1c-4b23-b63c-5bab1266ae39","Type":"ContainerStarted","Data":"fc49d6c01140c61629c3be2500566cf55a79d3138ceedd7bd8ce2fe6f140a27e"}
Nov 01 06:15:02.573168 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.572895    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg" event={"ID":"c8307b4c-2e1c-4b23-b63c-5bab1266ae39","Type":"ContainerStarted","Data":"6e86f9fee59cf688eda6594e26cc80988ebad775f826224ee2457c70aef53d94"}
Nov 01 06:15:02.578072 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.578045    5510 generic.go:356] "Generic (PLEG): container finished" podID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerID="c06c2fb6113894fd4f3f7555c5ac95218d1b329ed6cd0a5ba4e233f1d09c7be0" exitCode=60
Nov 01 06:15:02.578198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.578159    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" event={"ID":"8ccc4346-1335-4c04-a90b-266c8aaaddf1","Type":"ContainerDied","Data":"c06c2fb6113894fd4f3f7555c5ac95218d1b329ed6cd0a5ba4e233f1d09c7be0"}
Nov 01 06:15:02.578523 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.578494    5510 scope.go:122] "RemoveContainer" containerID="c06c2fb6113894fd4f3f7555c5ac95218d1b329ed6cd0a5ba4e233f1d09c7be0"
Nov 01 06:15:02.580461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.580441    5510 generic.go:356] "Generic (PLEG): container finished" podID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerID="527c731d19aaee8fd6e2a08b26d9e7e35fafd33206fae46d85c7106311a7b192" exitCode=60
Nov 01 06:15:02.580571 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.580470    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" event={"ID":"d2213f14-67e4-40ad-967c-2b7979b093fc","Type":"ContainerDied","Data":"527c731d19aaee8fd6e2a08b26d9e7e35fafd33206fae46d85c7106311a7b192"}
Nov 01 06:15:02.580748 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.580733    5510 scope.go:122] "RemoveContainer" containerID="527c731d19aaee8fd6e2a08b26d9e7e35fafd33206fae46d85c7106311a7b192"
Nov 01 06:15:02.801051 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:02.801003    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg" podStartSLOduration=2.800988213 podStartE2EDuration="2.800988213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:02.800241419 +0000 UTC m=+1739.678214829" watchObservedRunningTime="2026-11-01 06:15:02.800988213 +0000 UTC m=+1739.678961622"
Nov 01 06:15:03.588806 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.588770    5510 generic.go:356] "Generic (PLEG): container finished" podID="c8307b4c-2e1c-4b23-b63c-5bab1266ae39" containerID="fc49d6c01140c61629c3be2500566cf55a79d3138ceedd7bd8ce2fe6f140a27e" exitCode=0
Nov 01 06:15:03.589251 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.588859    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg" event={"ID":"c8307b4c-2e1c-4b23-b63c-5bab1266ae39","Type":"ContainerDied","Data":"fc49d6c01140c61629c3be2500566cf55a79d3138ceedd7bd8ce2fe6f140a27e"}
Nov 01 06:15:03.590798 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.590775    5510 generic.go:356] "Generic (PLEG): container finished" podID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerID="fe1f7d8139ab1c30238ddf7f2eb5f1fb265794478448087dbeb0e95cb56bfa12" exitCode=60
Nov 01 06:15:03.590941 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.590859    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" event={"ID":"8ccc4346-1335-4c04-a90b-266c8aaaddf1","Type":"ContainerDied","Data":"fe1f7d8139ab1c30238ddf7f2eb5f1fb265794478448087dbeb0e95cb56bfa12"}
Nov 01 06:15:03.590941 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.590915    5510 scope.go:122] "RemoveContainer" containerID="c06c2fb6113894fd4f3f7555c5ac95218d1b329ed6cd0a5ba4e233f1d09c7be0"
Nov 01 06:15:03.591199 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.591179    5510 scope.go:122] "RemoveContainer" containerID="fe1f7d8139ab1c30238ddf7f2eb5f1fb265794478448087dbeb0e95cb56bfa12"
Nov 01 06:15:03.591557 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:03.591526    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9_odh-ai-gateway-infra(8ccc4346-1335-4c04-a90b-266c8aaaddf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1"
Nov 01 06:15:03.592790 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.592770    5510 generic.go:356] "Generic (PLEG): container finished" podID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerID="a2f8881da3d11d4ae7b1ca87be550e8330c02d7536ef0304363d1408cbfe093b" exitCode=60
Nov 01 06:15:03.592905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.592797    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" event={"ID":"d2213f14-67e4-40ad-967c-2b7979b093fc","Type":"ContainerDied","Data":"a2f8881da3d11d4ae7b1ca87be550e8330c02d7536ef0304363d1408cbfe093b"}
Nov 01 06:15:03.593120 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.593104    5510 scope.go:122] "RemoveContainer" containerID="a2f8881da3d11d4ae7b1ca87be550e8330c02d7536ef0304363d1408cbfe093b"
Nov 01 06:15:03.593396 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:03.593376    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56_odh-ai-gateway-infra(d2213f14-67e4-40ad-967c-2b7979b093fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc"
Nov 01 06:15:03.594627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.594608    5510 generic.go:356] "Generic (PLEG): container finished" podID="e4cb781f-68e6-4468-86d7-df65be902d09" containerID="2e85dd2f722d31741de19fef67701266dfd4ccd97b2bae9fdfb6c22c34a266af" exitCode=60
Nov 01 06:15:03.594697 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.594643    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" event={"ID":"e4cb781f-68e6-4468-86d7-df65be902d09","Type":"ContainerDied","Data":"2e85dd2f722d31741de19fef67701266dfd4ccd97b2bae9fdfb6c22c34a266af"}
Nov 01 06:15:03.594897 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.594883    5510 scope.go:122] "RemoveContainer" containerID="2e85dd2f722d31741de19fef67701266dfd4ccd97b2bae9fdfb6c22c34a266af"
Nov 01 06:15:03.595136 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:03.595117    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs_odh-ai-gateway-infra(e4cb781f-68e6-4468-86d7-df65be902d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" podUID="e4cb781f-68e6-4468-86d7-df65be902d09"
Nov 01 06:15:03.606739 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.606719    5510 scope.go:122] "RemoveContainer" containerID="527c731d19aaee8fd6e2a08b26d9e7e35fafd33206fae46d85c7106311a7b192"
Nov 01 06:15:03.619533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:03.619517    5510 scope.go:122] "RemoveContainer" containerID="b71fc3cb28f8548476b436961f58f21fa577ca3c3a3f6d47c8cc09701717b829"
Nov 01 06:15:04.603553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.603516    5510 scope.go:122] "RemoveContainer" containerID="2e85dd2f722d31741de19fef67701266dfd4ccd97b2bae9fdfb6c22c34a266af"
Nov 01 06:15:04.604087 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:04.603864    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs_odh-ai-gateway-infra(e4cb781f-68e6-4468-86d7-df65be902d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" podUID="e4cb781f-68e6-4468-86d7-df65be902d09"
Nov 01 06:15:04.605239 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.605212    5510 scope.go:122] "RemoveContainer" containerID="fe1f7d8139ab1c30238ddf7f2eb5f1fb265794478448087dbeb0e95cb56bfa12"
Nov 01 06:15:04.605501 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:04.605479    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9_odh-ai-gateway-infra(8ccc4346-1335-4c04-a90b-266c8aaaddf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1"
Nov 01 06:15:04.607142 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.606916    5510 scope.go:122] "RemoveContainer" containerID="a2f8881da3d11d4ae7b1ca87be550e8330c02d7536ef0304363d1408cbfe093b"
Nov 01 06:15:04.607194 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:04.607144    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56_odh-ai-gateway-infra(d2213f14-67e4-40ad-967c-2b7979b093fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc"
Nov 01 06:15:04.766467 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.766439    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:04.847739 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.847702    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-openshift-ca\") pod \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") "
Nov 01 06:15:04.847930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.847794    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-xks-ca\") pod \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") "
Nov 01 06:15:04.847930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.847913    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-kube-api-access-6qcck\" (UniqueName: \"kubernetes.io/projected/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-kube-api-access-6qcck\") pod \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\" (UID: \"c8307b4c-2e1c-4b23-b63c-5bab1266ae39\") "
Nov 01 06:15:04.848115 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.848088    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-openshift-ca" pod "c8307b4c-2e1c-4b23-b63c-5bab1266ae39" (UID: "c8307b4c-2e1c-4b23-b63c-5bab1266ae39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:04.848413 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.848391    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:04.849938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.849912    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-xks-ca" pod "c8307b4c-2e1c-4b23-b63c-5bab1266ae39" (UID: "c8307b4c-2e1c-4b23-b63c-5bab1266ae39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:15:04.850164 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.850139    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-kube-api-access-6qcck" pod "c8307b4c-2e1c-4b23-b63c-5bab1266ae39" (UID: "c8307b4c-2e1c-4b23-b63c-5bab1266ae39"). InnerVolumeSpecName "kube-api-access-6qcck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:04.949237 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.949202    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:04.949237 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:04.949232    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qcck\" (UniqueName: \"kubernetes.io/projected/c8307b4c-2e1c-4b23-b63c-5bab1266ae39-kube-api-access-6qcck\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:05.615125 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:05.615049    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"
Nov 01 06:15:05.615492 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:05.615046    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg" event={"ID":"c8307b4c-2e1c-4b23-b63c-5bab1266ae39","Type":"ContainerDied","Data":"6e86f9fee59cf688eda6594e26cc80988ebad775f826224ee2457c70aef53d94"}
Nov 01 06:15:05.615492 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:05.615166    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e86f9fee59cf688eda6594e26cc80988ebad775f826224ee2457c70aef53d94"
Nov 01 06:15:11.693442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.693412    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"]
Nov 01 06:15:11.694151 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.694135    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c8307b4c-2e1c-4b23-b63c-5bab1266ae39" containerName="revoke-keys"
Nov 01 06:15:11.694207 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.694153    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8307b4c-2e1c-4b23-b63c-5bab1266ae39" containerName="revoke-keys"
Nov 01 06:15:11.694284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.694274    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8307b4c-2e1c-4b23-b63c-5bab1266ae39" containerName="revoke-keys"
Nov 01 06:15:11.697944 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.697929    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.751278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.751227    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:15:11.803239 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.803211    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"]
Nov 01 06:15:11.834627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.834589    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.834812 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.834696    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.834916 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.834878    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7sdw\" (UniqueName: \"kubernetes.io/projected/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-kube-api-access-v7sdw\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.936455 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.936418    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.936639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.936482    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.936639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.936587    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7sdw\" (UniqueName: \"kubernetes.io/projected/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-kube-api-access-v7sdw\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.937240 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.937215    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.938996 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.938973    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:11.999686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:11.999653    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7sdw\" (UniqueName: \"kubernetes.io/projected/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-kube-api-access-v7sdw\") pod \"maas-api-revoke-sub-d2abe5b5-v6qfz\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:12.011251 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:12.011224    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:12.404137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:12.404105    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"]
Nov 01 06:15:12.405720 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:15:12.405697    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d968108_8c39_46cb_a9a2_ea28fe7d4ffc.slice/crio-edce0e787f319194571fd0dbc4450ccbf7946f3c535d5a30406f19e8771c2a36 WatchSource:0}: Error finding container edce0e787f319194571fd0dbc4450ccbf7946f3c535d5a30406f19e8771c2a36: Status 404 returned error can't find the container with id edce0e787f319194571fd0dbc4450ccbf7946f3c535d5a30406f19e8771c2a36
Nov 01 06:15:12.660388 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:12.660302    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz" event={"ID":"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc","Type":"ContainerStarted","Data":"edce0e787f319194571fd0dbc4450ccbf7946f3c535d5a30406f19e8771c2a36"}
Nov 01 06:15:13.667840 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:13.667727    5510 generic.go:356] "Generic (PLEG): container finished" podID="2d968108-8c39-46cb-a9a2-ea28fe7d4ffc" containerID="802c064b4d8738459ba90da61f8d83953fb07d9848daab4623c6992978be98af" exitCode=0
Nov 01 06:15:13.667840 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:13.667806    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz" event={"ID":"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc","Type":"ContainerDied","Data":"802c064b4d8738459ba90da61f8d83953fb07d9848daab4623c6992978be98af"}
Nov 01 06:15:14.820450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.820418    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:14.868996 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.868968    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-xks-ca\") pod \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") "
Nov 01 06:15:14.869148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.869022    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-openshift-ca\") pod \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") "
Nov 01 06:15:14.869148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.869065    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-kube-api-access-v7sdw\" (UniqueName: \"kubernetes.io/projected/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-kube-api-access-v7sdw\") pod \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\" (UID: \"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc\") "
Nov 01 06:15:14.869403 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.869375    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-openshift-ca" pod "2d968108-8c39-46cb-a9a2-ea28fe7d4ffc" (UID: "2d968108-8c39-46cb-a9a2-ea28fe7d4ffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:14.869480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.869465    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.871284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.871261    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-xks-ca" pod "2d968108-8c39-46cb-a9a2-ea28fe7d4ffc" (UID: "2d968108-8c39-46cb-a9a2-ea28fe7d4ffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:15:14.871372 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.871353    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-kube-api-access-v7sdw" pod "2d968108-8c39-46cb-a9a2-ea28fe7d4ffc" (UID: "2d968108-8c39-46cb-a9a2-ea28fe7d4ffc"). InnerVolumeSpecName "kube-api-access-v7sdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:14.970922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.970888    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7sdw\" (UniqueName: \"kubernetes.io/projected/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-kube-api-access-v7sdw\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.970922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:14.970923    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:15.229050 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:15.228966    5510 scope.go:122] "RemoveContainer" containerID="fe1f7d8139ab1c30238ddf7f2eb5f1fb265794478448087dbeb0e95cb56bfa12"
Nov 01 06:15:15.682896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:15.682865    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" event={"ID":"8ccc4346-1335-4c04-a90b-266c8aaaddf1","Type":"ContainerStarted","Data":"6f5d01a3714d3238652467679552739d63500310b729aec4d30f7fd847b3d222"}
Nov 01 06:15:15.684407 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:15.684379    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz" event={"ID":"2d968108-8c39-46cb-a9a2-ea28fe7d4ffc","Type":"ContainerDied","Data":"edce0e787f319194571fd0dbc4450ccbf7946f3c535d5a30406f19e8771c2a36"}
Nov 01 06:15:15.684407 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:15.684397    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"
Nov 01 06:15:15.684565 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:15.684422    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edce0e787f319194571fd0dbc4450ccbf7946f3c535d5a30406f19e8771c2a36"
Nov 01 06:15:16.229539 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:16.229512    5510 scope.go:122] "RemoveContainer" containerID="a2f8881da3d11d4ae7b1ca87be550e8330c02d7536ef0304363d1408cbfe093b"
Nov 01 06:15:16.692949 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:16.692915    5510 generic.go:356] "Generic (PLEG): container finished" podID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerID="6f5d01a3714d3238652467679552739d63500310b729aec4d30f7fd847b3d222" exitCode=60
Nov 01 06:15:16.693122 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:16.692983    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" event={"ID":"8ccc4346-1335-4c04-a90b-266c8aaaddf1","Type":"ContainerDied","Data":"6f5d01a3714d3238652467679552739d63500310b729aec4d30f7fd847b3d222"}
Nov 01 06:15:16.693122 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:16.693051    5510 scope.go:122] "RemoveContainer" containerID="fe1f7d8139ab1c30238ddf7f2eb5f1fb265794478448087dbeb0e95cb56bfa12"
Nov 01 06:15:16.693418 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:16.693399    5510 scope.go:122] "RemoveContainer" containerID="6f5d01a3714d3238652467679552739d63500310b729aec4d30f7fd847b3d222"
Nov 01 06:15:16.693769 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:16.693742    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9_odh-ai-gateway-infra(8ccc4346-1335-4c04-a90b-266c8aaaddf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1"
Nov 01 06:15:16.695670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:16.695647    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" event={"ID":"d2213f14-67e4-40ad-967c-2b7979b093fc","Type":"ContainerStarted","Data":"46c56efe06100c0814329064a7072516a152e71a12641fd91fefac5ecfce1b22"}
Nov 01 06:15:16.808626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:16.808550    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"]
Nov 01 06:15:17.545620 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:17.545587    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:15:17.706031 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:17.706004    5510 generic.go:356] "Generic (PLEG): container finished" podID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerID="46c56efe06100c0814329064a7072516a152e71a12641fd91fefac5ecfce1b22" exitCode=60
Nov 01 06:15:17.706145 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:17.706080    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" event={"ID":"d2213f14-67e4-40ad-967c-2b7979b093fc","Type":"ContainerDied","Data":"46c56efe06100c0814329064a7072516a152e71a12641fd91fefac5ecfce1b22"}
Nov 01 06:15:17.706237 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:17.706153    5510 scope.go:122] "RemoveContainer" containerID="a2f8881da3d11d4ae7b1ca87be550e8330c02d7536ef0304363d1408cbfe093b"
Nov 01 06:15:17.706519 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:17.706504    5510 scope.go:122] "RemoveContainer" containerID="46c56efe06100c0814329064a7072516a152e71a12641fd91fefac5ecfce1b22"
Nov 01 06:15:17.706906 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:17.706878    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56_odh-ai-gateway-infra(d2213f14-67e4-40ad-967c-2b7979b093fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc"
Nov 01 06:15:17.951110 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:17.951067    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"]
Nov 01 06:15:18.025937 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.025913    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:18.139440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.139359    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ccc4346-1335-4c04-a90b-266c8aaaddf1-kube-api-access-9vhsj\" (UniqueName: \"kubernetes.io/projected/8ccc4346-1335-4c04-a90b-266c8aaaddf1-kube-api-access-9vhsj\") pod \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\" (UID: \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\") "
Nov 01 06:15:18.139594 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.139485    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccc4346-1335-4c04-a90b-266c8aaaddf1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccc4346-1335-4c04-a90b-266c8aaaddf1-maas-api-service-ca\") pod \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\" (UID: \"8ccc4346-1335-4c04-a90b-266c8aaaddf1\") "
Nov 01 06:15:18.140564 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.140146    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccc4346-1335-4c04-a90b-266c8aaaddf1-maas-api-service-ca" pod "8ccc4346-1335-4c04-a90b-266c8aaaddf1" (UID: "8ccc4346-1335-4c04-a90b-266c8aaaddf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:18.140564 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.140324    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccc4346-1335-4c04-a90b-266c8aaaddf1-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:18.142410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.142377    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccc4346-1335-4c04-a90b-266c8aaaddf1-kube-api-access-9vhsj" pod "8ccc4346-1335-4c04-a90b-266c8aaaddf1" (UID: "8ccc4346-1335-4c04-a90b-266c8aaaddf1"). InnerVolumeSpecName "kube-api-access-9vhsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:18.235852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.231521    5510 scope.go:122] "RemoveContainer" containerID="2e85dd2f722d31741de19fef67701266dfd4ccd97b2bae9fdfb6c22c34a266af"
Nov 01 06:15:18.249063 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.248816    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vhsj\" (UniqueName: \"kubernetes.io/projected/8ccc4346-1335-4c04-a90b-266c8aaaddf1-kube-api-access-9vhsj\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:18.714791 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.714758    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"
Nov 01 06:15:18.714791 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.714771    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9" event={"ID":"8ccc4346-1335-4c04-a90b-266c8aaaddf1","Type":"ContainerDied","Data":"46b09f11c078db5b2e4405982e943030d51639925674ce077860ef68b8a954f7"}
Nov 01 06:15:18.715298 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.714854    5510 scope.go:122] "RemoveContainer" containerID="6f5d01a3714d3238652467679552739d63500310b729aec4d30f7fd847b3d222"
Nov 01 06:15:18.719003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:18.718978    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" event={"ID":"e4cb781f-68e6-4468-86d7-df65be902d09","Type":"ContainerStarted","Data":"548fb222cc2ca7dab54909e190e3654dbdd029da0b66ae30bb66c796044759a2"}
Nov 01 06:15:19.022107 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.022083    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:19.060674 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.060579    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2213f14-67e4-40ad-967c-2b7979b093fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2213f14-67e4-40ad-967c-2b7979b093fc-maas-api-service-ca\") pod \"d2213f14-67e4-40ad-967c-2b7979b093fc\" (UID: \"d2213f14-67e4-40ad-967c-2b7979b093fc\") "
Nov 01 06:15:19.060882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.060688    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2213f14-67e4-40ad-967c-2b7979b093fc-kube-api-access-mx6fw\" (UniqueName: \"kubernetes.io/projected/d2213f14-67e4-40ad-967c-2b7979b093fc-kube-api-access-mx6fw\") pod \"d2213f14-67e4-40ad-967c-2b7979b093fc\" (UID: \"d2213f14-67e4-40ad-967c-2b7979b093fc\") "
Nov 01 06:15:19.061036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.061013    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2213f14-67e4-40ad-967c-2b7979b093fc-maas-api-service-ca" pod "d2213f14-67e4-40ad-967c-2b7979b093fc" (UID: "d2213f14-67e4-40ad-967c-2b7979b093fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:19.061249 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.061233    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2213f14-67e4-40ad-967c-2b7979b093fc-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:19.062997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.062971    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2213f14-67e4-40ad-967c-2b7979b093fc-kube-api-access-mx6fw" pod "d2213f14-67e4-40ad-967c-2b7979b093fc" (UID: "d2213f14-67e4-40ad-967c-2b7979b093fc"). InnerVolumeSpecName "kube-api-access-mx6fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:19.119438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.119401    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"]
Nov 01 06:15:19.126709 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.126671    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891895-gcxw9"]
Nov 01 06:15:19.194852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.192796    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx6fw\" (UniqueName: \"kubernetes.io/projected/d2213f14-67e4-40ad-967c-2b7979b093fc-kube-api-access-mx6fw\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:19.255053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.255018    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" path="/var/lib/kubelet/pods/8ccc4346-1335-4c04-a90b-266c8aaaddf1/volumes"
Nov 01 06:15:19.727708 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.727680    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"
Nov 01 06:15:19.727708 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.727703    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56" event={"ID":"d2213f14-67e4-40ad-967c-2b7979b093fc","Type":"ContainerDied","Data":"008ecaa44b581823a5524d51332a234ea1923c60dbaea661fee2aefc0d19f08a"}
Nov 01 06:15:19.728310 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.727778    5510 scope.go:122] "RemoveContainer" containerID="46c56efe06100c0814329064a7072516a152e71a12641fd91fefac5ecfce1b22"
Nov 01 06:15:19.730174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.730000    5510 generic.go:356] "Generic (PLEG): container finished" podID="e4cb781f-68e6-4468-86d7-df65be902d09" containerID="548fb222cc2ca7dab54909e190e3654dbdd029da0b66ae30bb66c796044759a2" exitCode=60
Nov 01 06:15:19.730174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.730080    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" event={"ID":"e4cb781f-68e6-4468-86d7-df65be902d09","Type":"ContainerDied","Data":"548fb222cc2ca7dab54909e190e3654dbdd029da0b66ae30bb66c796044759a2"}
Nov 01 06:15:19.731061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.731043    5510 scope.go:122] "RemoveContainer" containerID="548fb222cc2ca7dab54909e190e3654dbdd029da0b66ae30bb66c796044759a2"
Nov 01 06:15:19.731367 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:19.731344    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs_odh-ai-gateway-infra(e4cb781f-68e6-4468-86d7-df65be902d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" podUID="e4cb781f-68e6-4468-86d7-df65be902d09"
Nov 01 06:15:19.738320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.738304    5510 scope.go:122] "RemoveContainer" containerID="2e85dd2f722d31741de19fef67701266dfd4ccd97b2bae9fdfb6c22c34a266af"
Nov 01 06:15:19.892044 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:19.892012    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"]
Nov 01 06:15:20.023891 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:20.023806    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"]
Nov 01 06:15:20.026352 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:20.026327    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891895-v2v56"]
Nov 01 06:15:20.711265 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:20.711229    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:15:21.015677 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.015647    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:21.119598 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.119561    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cb781f-68e6-4468-86d7-df65be902d09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb781f-68e6-4468-86d7-df65be902d09-maas-api-service-ca\") pod \"e4cb781f-68e6-4468-86d7-df65be902d09\" (UID: \"e4cb781f-68e6-4468-86d7-df65be902d09\") "
Nov 01 06:15:21.119776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.119691    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cb781f-68e6-4468-86d7-df65be902d09-kube-api-access-8t7t9\" (UniqueName: \"kubernetes.io/projected/e4cb781f-68e6-4468-86d7-df65be902d09-kube-api-access-8t7t9\") pod \"e4cb781f-68e6-4468-86d7-df65be902d09\" (UID: \"e4cb781f-68e6-4468-86d7-df65be902d09\") "
Nov 01 06:15:21.120083 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.120055    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cb781f-68e6-4468-86d7-df65be902d09-maas-api-service-ca" pod "e4cb781f-68e6-4468-86d7-df65be902d09" (UID: "e4cb781f-68e6-4468-86d7-df65be902d09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:21.122124 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.122096    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cb781f-68e6-4468-86d7-df65be902d09-kube-api-access-8t7t9" pod "e4cb781f-68e6-4468-86d7-df65be902d09" (UID: "e4cb781f-68e6-4468-86d7-df65be902d09"). InnerVolumeSpecName "kube-api-access-8t7t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:21.221727 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.221692    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb781f-68e6-4468-86d7-df65be902d09-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:21.221727 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.221719    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7t9\" (UniqueName: \"kubernetes.io/projected/e4cb781f-68e6-4468-86d7-df65be902d09-kube-api-access-8t7t9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:21.236064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.236035    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" path="/var/lib/kubelet/pods/d2213f14-67e4-40ad-967c-2b7979b093fc/volumes"
Nov 01 06:15:21.748890 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.748861    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"
Nov 01 06:15:21.749092 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.748862    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs" event={"ID":"e4cb781f-68e6-4468-86d7-df65be902d09","Type":"ContainerDied","Data":"32bdcc823eb340f85a814526fd96e7d5193f395874faef81b542c9a616d16cbe"}
Nov 01 06:15:21.749092 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.749002    5510 scope.go:122] "RemoveContainer" containerID="548fb222cc2ca7dab54909e190e3654dbdd029da0b66ae30bb66c796044759a2"
Nov 01 06:15:21.914441 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.914406    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"]
Nov 01 06:15:21.917203 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:21.917176    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891895-cbmbs"]
Nov 01 06:15:23.236576 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:23.236544    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" path="/var/lib/kubelet/pods/e4cb781f-68e6-4468-86d7-df65be902d09/volumes"
Nov 01 06:15:42.634334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:42.634297    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"]
Nov 01 06:15:45.666609 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:45.666576    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"]
Nov 01 06:15:47.546398 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:47.546361    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:15:47.635247 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.635174    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d" podUID="2b38c4e1-a37f-4e62-ac69-f998f082e937" containerName="payload-processing" containerID="cri-o://67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe" gracePeriod=25
Nov 01 06:15:47.892494 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.892471    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:15:47.922504 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.922475    5510 generic.go:356] "Generic (PLEG): container finished" podID="2b38c4e1-a37f-4e62-ac69-f998f082e937" containerID="67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe" exitCode=0
Nov 01 06:15:47.922627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.922507    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d" event={"ID":"2b38c4e1-a37f-4e62-ac69-f998f082e937","Type":"ContainerDied","Data":"67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe"}
Nov 01 06:15:47.922627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.922526    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"
Nov 01 06:15:47.922627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.922561    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d" event={"ID":"2b38c4e1-a37f-4e62-ac69-f998f082e937","Type":"ContainerDied","Data":"663dc5e222965e865d5b3d571eb919891aa7233c628b27c4112f5be2dcd17c51"}
Nov 01 06:15:47.922627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.922583    5510 scope.go:122] "RemoveContainer" containerID="67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe"
Nov 01 06:15:47.931973 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.931949    5510 scope.go:122] "RemoveContainer" containerID="67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe"
Nov 01 06:15:47.932221 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:47.932204    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe\": container with ID starting with 67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe not found: ID does not exist" containerID="67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe"
Nov 01 06:15:47.932278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.932230    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe"} err="failed to get container status \"67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe\": rpc error: code = NotFound desc = could not find container \"67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe\": container with ID starting with 67ba5cc15fa1eee9f7e7751a852f0e99e514754eb4ce12731742d4f5310e9ebe not found: ID does not exist"
Nov 01 06:15:47.994604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.994581    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-provider-credentials\") pod \"2b38c4e1-a37f-4e62-ac69-f998f082e937\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") "
Nov 01 06:15:47.994731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.994625    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-routing-overlay-volume\") pod \"2b38c4e1-a37f-4e62-ac69-f998f082e937\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") "
Nov 01 06:15:47.994731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.994675    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-kube-api-access-rjbch\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-kube-api-access-rjbch\") pod \"2b38c4e1-a37f-4e62-ac69-f998f082e937\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") "
Nov 01 06:15:47.994731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.994696    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-plugins-config-volume\") pod \"2b38c4e1-a37f-4e62-ac69-f998f082e937\" (UID: \"2b38c4e1-a37f-4e62-ac69-f998f082e937\") "
Nov 01 06:15:47.995108 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.995076    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-routing-overlay-volume" pod "2b38c4e1-a37f-4e62-ac69-f998f082e937" (UID: "2b38c4e1-a37f-4e62-ac69-f998f082e937"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:47.995239 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.995217    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-plugins-config-volume" pod "2b38c4e1-a37f-4e62-ac69-f998f082e937" (UID: "2b38c4e1-a37f-4e62-ac69-f998f082e937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:47.995477 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.995323    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:47.995477 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.995336    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b38c4e1-a37f-4e62-ac69-f998f082e937-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:47.997073 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.997051    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-kube-api-access-rjbch" pod "2b38c4e1-a37f-4e62-ac69-f998f082e937" (UID: "2b38c4e1-a37f-4e62-ac69-f998f082e937"). InnerVolumeSpecName "kube-api-access-rjbch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:47.997153 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:47.997069    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-provider-credentials" pod "2b38c4e1-a37f-4e62-ac69-f998f082e937" (UID: "2b38c4e1-a37f-4e62-ac69-f998f082e937"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:48.096606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:48.096539    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:48.096606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:48.096564    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjbch\" (UniqueName: \"kubernetes.io/projected/2b38c4e1-a37f-4e62-ac69-f998f082e937-kube-api-access-rjbch\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:48.343224 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:48.343195    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"]
Nov 01 06:15:48.347127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:48.347063    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6x7r2d"]
Nov 01 06:15:49.237031 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:49.237003    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b38c4e1-a37f-4e62-ac69-f998f082e937" path="/var/lib/kubelet/pods/2b38c4e1-a37f-4e62-ac69-f998f082e937/volumes"
Nov 01 06:15:50.667677 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.667635    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl" podUID="83a0dd79-6e25-42f3-917b-b19464471a50" containerName="payload-processing" containerID="cri-o://beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb" gracePeriod=25
Nov 01 06:15:50.699128 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:15:50.699107    5510 logging.go:55] [core] [Channel #4362 SubChannel #4363]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.99:50052", ServerName: "10.134.0.99:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.99:50052: connect: connection refused"
Nov 01 06:15:50.937100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.937042    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:15:50.945174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.945145    5510 generic.go:356] "Generic (PLEG): container finished" podID="83a0dd79-6e25-42f3-917b-b19464471a50" containerID="beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb" exitCode=0
Nov 01 06:15:50.945305 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.945186    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"
Nov 01 06:15:50.945305 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.945228    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl" event={"ID":"83a0dd79-6e25-42f3-917b-b19464471a50","Type":"ContainerDied","Data":"beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb"}
Nov 01 06:15:50.945382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.945310    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl" event={"ID":"83a0dd79-6e25-42f3-917b-b19464471a50","Type":"ContainerDied","Data":"3f8154e17f607fe12db7f88edc1b2bd364b7551f4be6414d7cf47429ef953315"}
Nov 01 06:15:50.945382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.945335    5510 scope.go:122] "RemoveContainer" containerID="beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb"
Nov 01 06:15:50.955166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.955147    5510 scope.go:122] "RemoveContainer" containerID="beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb"
Nov 01 06:15:50.955458 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:15:50.955431    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb\": container with ID starting with beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb not found: ID does not exist" containerID="beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb"
Nov 01 06:15:50.955561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:50.955464    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb"} err="failed to get container status \"beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb\": rpc error: code = NotFound desc = could not find container \"beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb\": container with ID starting with beffdbe27f83336d6e3beaf608951f844540801f4288738235e847b9a3f3d3fb not found: ID does not exist"
Nov 01 06:15:51.032614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.032579    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-routing-overlay-volume\") pod \"83a0dd79-6e25-42f3-917b-b19464471a50\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") "
Nov 01 06:15:51.032809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.032650    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-provider-credentials\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-provider-credentials\") pod \"83a0dd79-6e25-42f3-917b-b19464471a50\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") "
Nov 01 06:15:51.032809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.032690    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-plugins-config-volume\") pod \"83a0dd79-6e25-42f3-917b-b19464471a50\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") "
Nov 01 06:15:51.032809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.032755    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-kube-api-access-788x6\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-kube-api-access-788x6\") pod \"83a0dd79-6e25-42f3-917b-b19464471a50\" (UID: \"83a0dd79-6e25-42f3-917b-b19464471a50\") "
Nov 01 06:15:51.033122 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.033099    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-routing-overlay-volume" pod "83a0dd79-6e25-42f3-917b-b19464471a50" (UID: "83a0dd79-6e25-42f3-917b-b19464471a50"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:51.033211 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.033196    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-plugins-config-volume" pod "83a0dd79-6e25-42f3-917b-b19464471a50" (UID: "83a0dd79-6e25-42f3-917b-b19464471a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:51.033267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.033250    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:51.035167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.035136    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-provider-credentials" pod "83a0dd79-6e25-42f3-917b-b19464471a50" (UID: "83a0dd79-6e25-42f3-917b-b19464471a50"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:51.035167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.035157    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-kube-api-access-788x6" pod "83a0dd79-6e25-42f3-917b-b19464471a50" (UID: "83a0dd79-6e25-42f3-917b-b19464471a50"). InnerVolumeSpecName "kube-api-access-788x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:51.134943 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.134911    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:51.134943 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.134938    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0dd79-6e25-42f3-917b-b19464471a50-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:51.134943 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.134947    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788x6\" (UniqueName: \"kubernetes.io/projected/83a0dd79-6e25-42f3-917b-b19464471a50-kube-api-access-788x6\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:51.359991 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.359960    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"]
Nov 01 06:15:51.362539 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.362503    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl"]
Nov 01 06:15:51.699255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:51.699169    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-79bbncljl" podUID="83a0dd79-6e25-42f3-917b-b19464471a50" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.99:50052\" within 1s: context deadline exceeded"
Nov 01 06:15:53.237958 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:53.237926    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a0dd79-6e25-42f3-917b-b19464471a50" path="/var/lib/kubelet/pods/83a0dd79-6e25-42f3-917b-b19464471a50/volumes"
Nov 01 06:15:54.074244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074214    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"]
Nov 01 06:15:54.074788 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074776    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.074852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074790    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.074852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074801    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:15:54.074852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074806    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:15:54.074852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074816    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2d968108-8c39-46cb-a9a2-ea28fe7d4ffc" containerName="revoke-keys"
Nov 01 06:15:54.074852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074821    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d968108-8c39-46cb-a9a2-ea28fe7d4ffc" containerName="revoke-keys"
Nov 01 06:15:54.074852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074848    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2b38c4e1-a37f-4e62-ac69-f998f082e937" containerName="payload-processing"
Nov 01 06:15:54.074852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074853    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b38c4e1-a37f-4e62-ac69-f998f082e937" containerName="payload-processing"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074861    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074866    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074877    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074881    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074889    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074894    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074902    5510 cpu_manager.go:414] "containerMap: removing container" podUID="83a0dd79-6e25-42f3-917b-b19464471a50" containerName="payload-processing"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074906    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a0dd79-6e25-42f3-917b-b19464471a50" containerName="payload-processing"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074919    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074923    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074933    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074937    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074948    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.074952    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075031    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075038    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075044    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a0dd79-6e25-42f3-917b-b19464471a50" containerName="payload-processing"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075050    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075058    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:15:54.075061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075064    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.075606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075071    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cb781f-68e6-4468-86d7-df65be902d09" containerName="cleanup"
Nov 01 06:15:54.075606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075077    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:15:54.075606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075082    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d968108-8c39-46cb-a9a2-ea28fe7d4ffc" containerName="revoke-keys"
Nov 01 06:15:54.075606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.075090    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b38c4e1-a37f-4e62-ac69-f998f082e937" containerName="payload-processing"
Nov 01 06:15:54.078471 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.078454    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.173245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173205    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96g4b\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-kube-api-access-96g4b\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.173245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173373    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.173245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173451    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.173245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173482    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.173245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173539    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.173245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173570    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0b5119-4d95-42f1-a501-494364aeee64-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.173245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173609    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0b5119-4d95-42f1-a501-494364aeee64-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.174272 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173682    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.174272 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.173777    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.184705 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.184676    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"]
Nov 01 06:15:54.280546 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.280514    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.280958 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.280594    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.280958 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.280707    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96g4b\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-kube-api-access-96g4b\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281072 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.280967    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281072 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281043    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281083    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281088    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281115    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281170    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281213    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0b5119-4d95-42f1-a501-494364aeee64-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281262    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0b5119-4d95-42f1-a501-494364aeee64-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281498 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281406    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.281547 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.281508    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.282145 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.282123    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0b5119-4d95-42f1-a501-494364aeee64-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.283875 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.283848    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.284347 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.284326    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0b5119-4d95-42f1-a501-494364aeee64-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.341864 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.341766    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.342003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.341897    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96g4b\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-kube-api-access-96g4b\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.391810 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.391780    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:54.828159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.828127    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"]
Nov 01 06:15:54.828663 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:15:54.828638    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0b5119_4d95_42f1_a501_494364aeee64.slice/crio-268b8ad74c7f15ba14cf804f1838b21fcb99b63c64a731d5cf8bc450a111ebee WatchSource:0}: Error finding container 268b8ad74c7f15ba14cf804f1838b21fcb99b63c64a731d5cf8bc450a111ebee: Status 404 returned error can't find the container with id 268b8ad74c7f15ba14cf804f1838b21fcb99b63c64a731d5cf8bc450a111ebee
Nov 01 06:15:54.831190 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.831157    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:15:54.831361 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.831226    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:15:54.831361 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.831255    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:15:54.975720 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.975685    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" event={"ID":"0e0b5119-4d95-42f1-a501-494364aeee64","Type":"ContainerStarted","Data":"a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef"}
Nov 01 06:15:54.975926 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:54.975748    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" event={"ID":"0e0b5119-4d95-42f1-a501-494364aeee64","Type":"ContainerStarted","Data":"268b8ad74c7f15ba14cf804f1838b21fcb99b63c64a731d5cf8bc450a111ebee"}
Nov 01 06:15:55.098622 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:55.098530    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podStartSLOduration=1.098515662 podStartE2EDuration="1.098515662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:55.094815071 +0000 UTC m=+1791.972788506" watchObservedRunningTime="2026-11-01 06:15:55.098515662 +0000 UTC m=+1791.976489072"
Nov 01 06:15:55.397049 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:55.396894    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:15:55.397928 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:55.397896    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 06:15:55.398083 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:55.397997    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 06:15:56.393856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:56.392231    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 06:15:56.393856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:56.392293    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 06:15:57.393264 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:57.393229    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 06:15:57.393929 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:57.393869    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 06:15:58.394258 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:58.394218    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 06:15:58.394795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:58.394288    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 06:15:59.395849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:59.395192    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 06:15:59.395849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:15:59.395259    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 06:16:00.394301 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:00.394257    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 06:16:00.394460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:00.394324    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 06:16:01.392253 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:01.392222    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 06:16:01.392663 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:01.392276    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 06:16:02.396650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:02.396619    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:16:02.397136 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:02.397059    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:16:02.397700 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:02.397681    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:16:04.943998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:04.943959    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:16:09.749197 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:09.749161    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"]
Nov 01 06:16:09.749711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:09.749391    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy" containerID="cri-o://a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0" gracePeriod=30
Nov 01 06:16:09.800050 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:09.800018    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"]
Nov 01 06:16:09.800321 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:09.800296    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy" containerID="cri-o://0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925" gracePeriod=30
Nov 01 06:16:15.074656 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.074629    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:16:15.102003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.101980    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:16:15.202047 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202016    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-kube-api-access-bkf2r\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-kube-api-access-bkf2r\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.202047 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202051    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-token\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-token\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.202271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202087    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-data\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.202271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202137    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-podinfo\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.202271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202192    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-socket\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.202271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202229    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-token\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.202485 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202302    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istiod-ca-cert\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.202485 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202343    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-credential-socket\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.202485 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202399    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-certs\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.202485 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202457    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-envoy\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.202485 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202456    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-data" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.202765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202506    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-kube-api-access-8dwdn\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-kube-api-access-8dwdn\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.202765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202551    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-socket" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.202765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202564    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965e2e7d-daae-459d-9381-c88fd38dc2e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965e2e7d-daae-459d-9381-c88fd38dc2e3-istiod-ca-cert\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.202765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202596    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-socket\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.202765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202650    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-certs\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.202765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202675    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-credential-socket" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.202765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202681    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-podinfo\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.202765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202740    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-envoy\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.203400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202793    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-data\") pod \"965e2e7d-daae-459d-9381-c88fd38dc2e3\" (UID: \"965e2e7d-daae-459d-9381-c88fd38dc2e3\") "
Nov 01 06:16:15.203400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202865    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-credential-socket\") pod \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\" (UID: \"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90\") "
Nov 01 06:16:15.203400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.202986    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-certs" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.203400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203281    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istiod-ca-cert" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:16:15.203400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203335    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965e2e7d-daae-459d-9381-c88fd38dc2e3-istiod-ca-cert" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:16:15.203656 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203464    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-credential-socket" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.203656 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203649    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-data" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.203765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203665    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965e2e7d-daae-459d-9381-c88fd38dc2e3-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.203765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203687    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.203765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203703    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.203765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203717    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.203765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203748    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.203765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203762    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.204090 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203775    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.204090 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203821    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-socket" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.204090 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.203886    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-certs" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.204891 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.204868    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-kube-api-access-bkf2r" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "kube-api-access-bkf2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:15.205111 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.205081    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-podinfo" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:16:15.205223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.205203    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-token" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:15.205370 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.205352    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-envoy" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.206133 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.206109    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-kube-api-access-8dwdn" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "kube-api-access-8dwdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:15.206133 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.206120    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-token" pod "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" (UID: "3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:15.206628 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.206613    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-podinfo" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:16:15.206970 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.206955    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-envoy" pod "965e2e7d-daae-459d-9381-c88fd38dc2e3" (UID: "965e2e7d-daae-459d-9381-c88fd38dc2e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:15.241128 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.241099    5510 generic.go:356] "Generic (PLEG): container finished" podID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerID="0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925" exitCode=0
Nov 01 06:16:15.241263 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.241154    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"
Nov 01 06:16:15.241263 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.241177    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" event={"ID":"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90","Type":"ContainerDied","Data":"0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925"}
Nov 01 06:16:15.241263 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.241234    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk" event={"ID":"3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90","Type":"ContainerDied","Data":"42a3fa888de285639eaee33d333a58ee0c03ff3a937e39a133bbfcbaa4bcc850"}
Nov 01 06:16:15.241424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.241266    5510 scope.go:122] "RemoveContainer" containerID="0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925"
Nov 01 06:16:15.242896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.242870    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" event={"ID":"965e2e7d-daae-459d-9381-c88fd38dc2e3","Type":"ContainerDied","Data":"a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0"}
Nov 01 06:16:15.242896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.242891    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"
Nov 01 06:16:15.243071 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.242811    5510 generic.go:356] "Generic (PLEG): container finished" podID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerID="a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0" exitCode=0
Nov 01 06:16:15.243130 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.243100    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t" event={"ID":"965e2e7d-daae-459d-9381-c88fd38dc2e3","Type":"ContainerDied","Data":"f29e29eed0849daacc94ecc483a3b1ad9fe40dc47c5a22de851b66cfe591d0f2"}
Nov 01 06:16:15.264010 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.254670    5510 scope.go:122] "RemoveContainer" containerID="0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925"
Nov 01 06:16:15.264010 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:16:15.255159    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925\": container with ID starting with 0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925 not found: ID does not exist" containerID="0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925"
Nov 01 06:16:15.264010 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.255205    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925"} err="failed to get container status \"0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925\": rpc error: code = NotFound desc = could not find container \"0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925\": container with ID starting with 0c116d5b75d1d71dcfbe88de747c5904055101ec11645ee4b486f3c2ac122925 not found: ID does not exist"
Nov 01 06:16:15.264010 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.255228    5510 scope.go:122] "RemoveContainer" containerID="a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0"
Nov 01 06:16:15.266046 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.265945    5510 scope.go:122] "RemoveContainer" containerID="a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0"
Nov 01 06:16:15.266335 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:16:15.266315    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0\": container with ID starting with a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0 not found: ID does not exist" containerID="a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0"
Nov 01 06:16:15.266404 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.266340    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0"} err="failed to get container status \"a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0\": rpc error: code = NotFound desc = could not find container \"a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0\": container with ID starting with a280f406656204eb373d74dce1acb79761bba50a6ef36592d3ab168047361ab0 not found: ID does not exist"
Nov 01 06:16:15.304570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304494    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304519    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwdn\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-kube-api-access-8dwdn\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304535    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304551    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304564    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304676    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304705    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304721    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkf2r\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-kube-api-access-bkf2r\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304739    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965e2e7d-daae-459d-9381-c88fd38dc2e3-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304752    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.304925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.304766    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:15.533178 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.533145    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"]
Nov 01 06:16:15.539869 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.539804    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xvdnk"]
Nov 01 06:16:15.648938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.648863    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"]
Nov 01 06:16:15.655005 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:15.654982    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6gf5t"]
Nov 01 06:16:17.236690 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:17.236657    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" path="/var/lib/kubelet/pods/3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90/volumes"
Nov 01 06:16:17.237129 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:17.237115    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" path="/var/lib/kubelet/pods/965e2e7d-daae-459d-9381-c88fd38dc2e3/volumes"
Nov 01 06:16:17.545713 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:16:17.545629    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:16:44.414529 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.414495    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"]
Nov 01 06:16:44.415327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415304    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy"
Nov 01 06:16:44.415428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415329    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy"
Nov 01 06:16:44.415428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415346    5510 cpu_manager.go:414] "containerMap: removing container" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy"
Nov 01 06:16:44.415428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415352    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy"
Nov 01 06:16:44.415428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415365    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:16:44.415428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415372    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:16:44.415669 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415559    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0e5fb6-ad11-4bc6-a62b-f9810b72fb90" containerName="istio-proxy"
Nov 01 06:16:44.415669 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415576    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="965e2e7d-daae-459d-9381-c88fd38dc2e3" containerName="istio-proxy"
Nov 01 06:16:44.415669 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415593    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2213f14-67e4-40ad-967c-2b7979b093fc" containerName="cleanup"
Nov 01 06:16:44.415669 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.415607    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccc4346-1335-4c04-a90b-266c8aaaddf1" containerName="cleanup"
Nov 01 06:16:44.419226 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.419205    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.438034 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.438008    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b837176a-c74b-4203-85a9-acb1120c5f97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.438124 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.438045    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m88jh\" (UniqueName: \"kubernetes.io/projected/b837176a-c74b-4203-85a9-acb1120c5f97-kube-api-access-m88jh\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.438124 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.438069    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.438201 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.438168    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.471192 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.471165    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-71cea4-serving-cert\""
Nov 01 06:16:44.524300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.524265    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"]
Nov 01 06:16:44.539108 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.539074    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b837176a-c74b-4203-85a9-acb1120c5f97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.539284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.539121    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88jh\" (UniqueName: \"kubernetes.io/projected/b837176a-c74b-4203-85a9-acb1120c5f97-kube-api-access-m88jh\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.539284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.539248    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.539399 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.539306    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.539487 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:16:44.539471    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-serving-cert: secret "maas-api-e2e-shared-a-w0-71cea4-serving-cert" not found
Nov 01 06:16:44.539566 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:16:44.539554    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls podName:b837176a-c74b-4203-85a9-acb1120c5f97 nodeName:}" failed. No retries permitted until 2026-11-01 06:16:45.039532589 +0000 UTC m=+1841.917505992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls") pod "maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs" (UID: "b837176a-c74b-4203-85a9-acb1120c5f97") : secret "maas-api-e2e-shared-a-w0-71cea4-serving-cert" not found
Nov 01 06:16:44.539967 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.539942    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b837176a-c74b-4203-85a9-acb1120c5f97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.541919 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.541896    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:44.600340 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:44.600306    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m88jh\" (UniqueName: \"kubernetes.io/projected/b837176a-c74b-4203-85a9-acb1120c5f97-kube-api-access-m88jh\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:45.045225 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:45.045190    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:45.047873 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:45.047850    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:45.332563 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:45.332482    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:45.712445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:45.712417    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"]
Nov 01 06:16:45.713347 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:16:45.713321    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb837176a_c74b_4203_85a9_acb1120c5f97.slice/crio-8cec0b8f3eea8884d304a5dc3d5345dde9e7876bbd4a1fb3cc6c28b5b4cc34b2 WatchSource:0}: Error finding container 8cec0b8f3eea8884d304a5dc3d5345dde9e7876bbd4a1fb3cc6c28b5b4cc34b2: Status 404 returned error can't find the container with id 8cec0b8f3eea8884d304a5dc3d5345dde9e7876bbd4a1fb3cc6c28b5b4cc34b2
Nov 01 06:16:45.714963 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:45.714946    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:16:46.465681 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:46.465641    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs" event={"ID":"b837176a-c74b-4203-85a9-acb1120c5f97","Type":"ContainerStarted","Data":"8cec0b8f3eea8884d304a5dc3d5345dde9e7876bbd4a1fb3cc6c28b5b4cc34b2"}
Nov 01 06:16:47.494152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:47.494103    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs" event={"ID":"b837176a-c74b-4203-85a9-acb1120c5f97","Type":"ContainerStarted","Data":"01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d"}
Nov 01 06:16:47.494672 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:47.494220    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:16:47.544444 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:16:47.544409    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:16:47.611960 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:47.611909    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs" podStartSLOduration=2.7402477039999997 podStartE2EDuration="3.611895274s" totalImagesPullingTime="871.64757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:47.610887004 +0000 UTC m=+1844.488860416" watchObservedRunningTime="2026-11-01 06:16:47.611895274 +0000 UTC m=+1844.489868678"
Nov 01 06:16:50.872018 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:50.871979    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:16:53.507437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:16:53.507408    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:17:17.545842 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:17:17.545780    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:17:22.784950 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:22.784919    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"]
Nov 01 06:17:22.887623 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:22.887590    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"]
Nov 01 06:17:27.785648 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:27.785573    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn" podUID="4108c3c4-065e-4e2a-b9c1-f1d134357b9f" containerName="payload-processing" containerID="cri-o://8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe" gracePeriod=25
Nov 01 06:17:27.888447 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:27.888385    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg" podUID="67a331ec-2379-4453-906a-8cd6aa9996e0" containerName="payload-pre-processing" containerID="cri-o://627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07" gracePeriod=25
Nov 01 06:17:28.063490 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.063466    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:17:28.171686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.171662    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:17:28.183921 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.183887    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-plugins-config-volume\") pod \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\" (UID: \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\") "
Nov 01 06:17:28.184096 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.183953    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-kube-api-access-tbcm9\" (UniqueName: \"kubernetes.io/projected/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-kube-api-access-tbcm9\") pod \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\" (UID: \"4108c3c4-065e-4e2a-b9c1-f1d134357b9f\") "
Nov 01 06:17:28.184298 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.184275    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-plugins-config-volume" pod "4108c3c4-065e-4e2a-b9c1-f1d134357b9f" (UID: "4108c3c4-065e-4e2a-b9c1-f1d134357b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:17:28.184501 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.184477    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:28.186438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.186416    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-kube-api-access-tbcm9" pod "4108c3c4-065e-4e2a-b9c1-f1d134357b9f" (UID: "4108c3c4-065e-4e2a-b9c1-f1d134357b9f"). InnerVolumeSpecName "kube-api-access-tbcm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:28.285568 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.285536    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a331ec-2379-4453-906a-8cd6aa9996e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a331ec-2379-4453-906a-8cd6aa9996e0-plugins-config-volume\") pod \"67a331ec-2379-4453-906a-8cd6aa9996e0\" (UID: \"67a331ec-2379-4453-906a-8cd6aa9996e0\") "
Nov 01 06:17:28.285740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.285702    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a331ec-2379-4453-906a-8cd6aa9996e0-kube-api-access-4s9wd\" (UniqueName: \"kubernetes.io/projected/67a331ec-2379-4453-906a-8cd6aa9996e0-kube-api-access-4s9wd\") pod \"67a331ec-2379-4453-906a-8cd6aa9996e0\" (UID: \"67a331ec-2379-4453-906a-8cd6aa9996e0\") "
Nov 01 06:17:28.285982 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.285954    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a331ec-2379-4453-906a-8cd6aa9996e0-plugins-config-volume" pod "67a331ec-2379-4453-906a-8cd6aa9996e0" (UID: "67a331ec-2379-4453-906a-8cd6aa9996e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:17:28.286127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.286106    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbcm9\" (UniqueName: \"kubernetes.io/projected/4108c3c4-065e-4e2a-b9c1-f1d134357b9f-kube-api-access-tbcm9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:28.286127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.286129    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a331ec-2379-4453-906a-8cd6aa9996e0-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:28.287923 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.287902    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a331ec-2379-4453-906a-8cd6aa9996e0-kube-api-access-4s9wd" pod "67a331ec-2379-4453-906a-8cd6aa9996e0" (UID: "67a331ec-2379-4453-906a-8cd6aa9996e0"). InnerVolumeSpecName "kube-api-access-4s9wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:28.387273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.387178    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s9wd\" (UniqueName: \"kubernetes.io/projected/67a331ec-2379-4453-906a-8cd6aa9996e0-kube-api-access-4s9wd\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:28.751029 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.750993    5510 generic.go:356] "Generic (PLEG): container finished" podID="67a331ec-2379-4453-906a-8cd6aa9996e0" containerID="627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07" exitCode=0
Nov 01 06:17:28.751205 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.751038    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"
Nov 01 06:17:28.751205 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.751075    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg" event={"ID":"67a331ec-2379-4453-906a-8cd6aa9996e0","Type":"ContainerDied","Data":"627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07"}
Nov 01 06:17:28.751205 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.751129    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg" event={"ID":"67a331ec-2379-4453-906a-8cd6aa9996e0","Type":"ContainerDied","Data":"79736aa97b106505e34185cf632387209e7f654ceb0aad1757ae1bad023afec4"}
Nov 01 06:17:28.751205 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.751161    5510 scope.go:122] "RemoveContainer" containerID="627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07"
Nov 01 06:17:28.752614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.752593    5510 generic.go:356] "Generic (PLEG): container finished" podID="4108c3c4-065e-4e2a-b9c1-f1d134357b9f" containerID="8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe" exitCode=0
Nov 01 06:17:28.752714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.752639    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"
Nov 01 06:17:28.752714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.752678    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn" event={"ID":"4108c3c4-065e-4e2a-b9c1-f1d134357b9f","Type":"ContainerDied","Data":"8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe"}
Nov 01 06:17:28.752814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.752732    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn" event={"ID":"4108c3c4-065e-4e2a-b9c1-f1d134357b9f","Type":"ContainerDied","Data":"c3c2a365e6600bfeba310d9df4ef2855706ee800a53cd491b4ece4fdeda009ae"}
Nov 01 06:17:28.761368 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.761337    5510 scope.go:122] "RemoveContainer" containerID="627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07"
Nov 01 06:17:28.761688 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:17:28.761667    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07\": container with ID starting with 627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07 not found: ID does not exist" containerID="627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07"
Nov 01 06:17:28.761761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.761694    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07"} err="failed to get container status \"627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07\": rpc error: code = NotFound desc = could not find container \"627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07\": container with ID starting with 627f0dface07850e22e832394efc3e54d385fc60d15a8a710c579917c7fa0a07 not found: ID does not exist"
Nov 01 06:17:28.761761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.761712    5510 scope.go:122] "RemoveContainer" containerID="8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe"
Nov 01 06:17:28.770775 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.770758    5510 scope.go:122] "RemoveContainer" containerID="8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe"
Nov 01 06:17:28.771119 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:17:28.771101    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe\": container with ID starting with 8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe not found: ID does not exist" containerID="8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe"
Nov 01 06:17:28.771192 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.771125    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe"} err="failed to get container status \"8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe\": rpc error: code = NotFound desc = could not find container \"8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe\": container with ID starting with 8e59d4674e7a94fa1943390bce6e2267fbefcf1e8205a355ff454a10f79dbebe not found: ID does not exist"
Nov 01 06:17:28.915790 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.915756    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"]
Nov 01 06:17:28.921509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:28.921484    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec5596-5f9fbd966c-msdpn"]
Nov 01 06:17:29.030203 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:29.030120    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"]
Nov 01 06:17:29.033914 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:29.033890    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec5596-7b8b5d594c-lwxqg"]
Nov 01 06:17:29.236569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:29.236539    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4108c3c4-065e-4e2a-b9c1-f1d134357b9f" path="/var/lib/kubelet/pods/4108c3c4-065e-4e2a-b9c1-f1d134357b9f/volumes"
Nov 01 06:17:29.236962 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:29.236949    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a331ec-2379-4453-906a-8cd6aa9996e0" path="/var/lib/kubelet/pods/67a331ec-2379-4453-906a-8cd6aa9996e0/volumes"
Nov 01 06:17:32.579606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:17:32.579570    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:17:47.545189 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:17:47.545102    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:18:14.412341 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:14.412304    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:18:17.545135 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:18:17.545100    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:18:25.189263 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:25.189227    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"]
Nov 01 06:18:25.189712 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:25.189437    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy" containerID="cri-o://2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b" gracePeriod=30
Nov 01 06:18:30.451778 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.451753    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:18:30.550174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550142    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-certs\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550195    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-envoy\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550226    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ba1edb-8e44-4567-8936-6808f39ee230-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ba1edb-8e44-4567-8936-6808f39ee230-istiod-ca-cert\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550254    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-credential-socket\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550286    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-kube-api-access-brw7z\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-kube-api-access-brw7z\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550335    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-data\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550364    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10ba1edb-8e44-4567-8936-6808f39ee230-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ba1edb-8e44-4567-8936-6808f39ee230-istio-podinfo\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550438    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-istio-token\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-istio-token\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550509    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-socket\") pod \"10ba1edb-8e44-4567-8936-6808f39ee230\" (UID: \"10ba1edb-8e44-4567-8936-6808f39ee230\") "
Nov 01 06:18:30.550634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550422    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-certs" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.550911 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550580    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-credential-socket" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.550911 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550656    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-data" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.550911 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550677    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ba1edb-8e44-4567-8936-6808f39ee230-istiod-ca-cert" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:18:30.550911 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.550904    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-socket" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.551124 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.551104    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ba1edb-8e44-4567-8936-6808f39ee230-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.551166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.551130    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.551166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.551145    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.551166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.551158    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.551297 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.551172    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.552654 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.552634    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-envoy" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.552839 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.552807    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10ba1edb-8e44-4567-8936-6808f39ee230-istio-podinfo" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:18:30.553196 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.553169    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-istio-token" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:30.553293 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.553192    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-kube-api-access-brw7z" pod "10ba1edb-8e44-4567-8936-6808f39ee230" (UID: "10ba1edb-8e44-4567-8936-6808f39ee230"). InnerVolumeSpecName "kube-api-access-brw7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:30.652259 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.652229    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ba1edb-8e44-4567-8936-6808f39ee230-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.652259 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.652256    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brw7z\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-kube-api-access-brw7z\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.652259 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.652267    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ba1edb-8e44-4567-8936-6808f39ee230-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.652541 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:30.652276    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ba1edb-8e44-4567-8936-6808f39ee230-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:31.279721 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.279687    5510 generic.go:356] "Generic (PLEG): container finished" podID="10ba1edb-8e44-4567-8936-6808f39ee230" containerID="2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b" exitCode=0
Nov 01 06:18:31.279881 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.279738    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"
Nov 01 06:18:31.279881 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.279761    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" event={"ID":"10ba1edb-8e44-4567-8936-6808f39ee230","Type":"ContainerDied","Data":"2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b"}
Nov 01 06:18:31.279881 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.279814    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb" event={"ID":"10ba1edb-8e44-4567-8936-6808f39ee230","Type":"ContainerDied","Data":"928365e34cc32c36f29e959e95309db901aed9e27d3148a7eea2abb8516189ff"}
Nov 01 06:18:31.279881 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.279862    5510 scope.go:122] "RemoveContainer" containerID="2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b"
Nov 01 06:18:31.289763 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.289735    5510 scope.go:122] "RemoveContainer" containerID="2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b"
Nov 01 06:18:31.290063 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:18:31.290034    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b\": container with ID starting with 2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b not found: ID does not exist" containerID="2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b"
Nov 01 06:18:31.290142 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.290074    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b"} err="failed to get container status \"2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b\": rpc error: code = NotFound desc = could not find container \"2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b\": container with ID starting with 2879afc6b890bcea089e07c44d006853975b3014d0f84c0ba35ebfda30cc834b not found: ID does not exist"
Nov 01 06:18:31.496806 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.496727    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"]
Nov 01 06:18:31.546187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:31.546156    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-bpdwb"]
Nov 01 06:18:33.236285 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:33.236252    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" path="/var/lib/kubelet/pods/10ba1edb-8e44-4567-8936-6808f39ee230/volumes"
Nov 01 06:18:40.336805 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:40.336763    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"]
Nov 01 06:18:40.337332 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:40.337035    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy" containerID="cri-o://c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5" gracePeriod=30
Nov 01 06:18:41.623586 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:41.623547    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:18:41.623981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:41.623616    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:18:45.589313 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.589290    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:18:45.728372 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728341    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-token\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-token\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728396    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-kube-api-access-nql5k\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-kube-api-access-nql5k\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728425    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-envoy\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728479    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-socket\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728511    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-credential-socket\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728793 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728641    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-certs\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728793 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728684    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-podinfo\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728793 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728741    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab11984-22df-4438-afa2-a4bed63a76e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab11984-22df-4438-afa2-a4bed63a76e5-istiod-ca-cert\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728808    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-credential-socket" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.728997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728818    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-data\") pod \"8ab11984-22df-4438-afa2-a4bed63a76e5\" (UID: \"8ab11984-22df-4438-afa2-a4bed63a76e5\") "
Nov 01 06:18:45.728997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728921    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-socket" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.728997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.728940    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-certs" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.729460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.729383    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.729460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.729408    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.729460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.729422    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.729460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.729442    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab11984-22df-4438-afa2-a4bed63a76e5-istiod-ca-cert" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:18:45.729711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.729505    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-data" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.730939 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.730912    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-kube-api-access-nql5k" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "kube-api-access-nql5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:45.731047 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.730946    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-token" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:45.731129 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.731111    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-envoy" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.731362 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.731346    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-podinfo" pod "8ab11984-22df-4438-afa2-a4bed63a76e5" (UID: "8ab11984-22df-4438-afa2-a4bed63a76e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:18:45.830946 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.830915    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.830946 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.830946    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab11984-22df-4438-afa2-a4bed63a76e5-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.831156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.830962    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.831156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.830978    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.831156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.830993    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nql5k\" (UniqueName: \"kubernetes.io/projected/8ab11984-22df-4438-afa2-a4bed63a76e5-kube-api-access-nql5k\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.831156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:45.831009    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab11984-22df-4438-afa2-a4bed63a76e5-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:46.375400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.375363    5510 generic.go:356] "Generic (PLEG): container finished" podID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerID="c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5" exitCode=0
Nov 01 06:18:46.375587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.375408    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"
Nov 01 06:18:46.375587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.375448    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" event={"ID":"8ab11984-22df-4438-afa2-a4bed63a76e5","Type":"ContainerDied","Data":"c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5"}
Nov 01 06:18:46.375587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.375500    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx" event={"ID":"8ab11984-22df-4438-afa2-a4bed63a76e5","Type":"ContainerDied","Data":"b88cdc58a9a60ed77f6da71e9b4388e1f15078139f95b2d5c9a0827608f82c94"}
Nov 01 06:18:46.375587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.375524    5510 scope.go:122] "RemoveContainer" containerID="c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5"
Nov 01 06:18:46.386186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.386169    5510 scope.go:122] "RemoveContainer" containerID="c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5"
Nov 01 06:18:46.386444 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:18:46.386427    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5\": container with ID starting with c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5 not found: ID does not exist" containerID="c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5"
Nov 01 06:18:46.386495 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.386454    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5"} err="failed to get container status \"c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5\": rpc error: code = NotFound desc = could not find container \"c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5\": container with ID starting with c0c5dc8e9b19fda5a184c72e3b593289865258872d478c634d35874a3af700d5 not found: ID does not exist"
Nov 01 06:18:46.588661 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.588627    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"]
Nov 01 06:18:46.638015 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:46.637931    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-4s9gx"]
Nov 01 06:18:47.236311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:47.236276    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" path="/var/lib/kubelet/pods/8ab11984-22df-4438-afa2-a4bed63a76e5/volumes"
Nov 01 06:18:47.544689 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:18:47.544605    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:18:48.529310 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:18:48.529270    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:19:16.397741 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:16.397647    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:19:17.544950 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:19:17.544914    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:19:43.552053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:43.552006    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"]
Nov 01 06:19:47.545879 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:19:47.545818    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:19:48.552701 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.552665    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk" podUID="9e374f81-bec1-477b-8ee1-e5631744acb0" containerName="payload-processing" containerID="cri-o://7fe5d34b0b2d31cb67f7ea2b57af92dbf640e42d49affd489986cc6a2f0f550d" gracePeriod=25
Nov 01 06:19:48.819769 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.819697    5510 generic.go:356] "Generic (PLEG): container finished" podID="9e374f81-bec1-477b-8ee1-e5631744acb0" containerID="7fe5d34b0b2d31cb67f7ea2b57af92dbf640e42d49affd489986cc6a2f0f550d" exitCode=0
Nov 01 06:19:48.819988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.819777    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk" event={"ID":"9e374f81-bec1-477b-8ee1-e5631744acb0","Type":"ContainerDied","Data":"7fe5d34b0b2d31cb67f7ea2b57af92dbf640e42d49affd489986cc6a2f0f550d"}
Nov 01 06:19:48.819988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.819845    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk" event={"ID":"9e374f81-bec1-477b-8ee1-e5631744acb0","Type":"ContainerDied","Data":"1b56ed1ebf48febc44349245eafd77bef92029ee7b97f6ee2791538b7505cdf4"}
Nov 01 06:19:48.819988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.819866    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b56ed1ebf48febc44349245eafd77bef92029ee7b97f6ee2791538b7505cdf4"
Nov 01 06:19:48.830577 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.830555    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:19:48.865892 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.865869    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-provider-credentials\") pod \"9e374f81-bec1-477b-8ee1-e5631744acb0\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") "
Nov 01 06:19:48.866013 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.865915    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-kube-api-access-x92b9\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-kube-api-access-x92b9\") pod \"9e374f81-bec1-477b-8ee1-e5631744acb0\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") "
Nov 01 06:19:48.866013 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.865941    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-plugins-config-volume\") pod \"9e374f81-bec1-477b-8ee1-e5631744acb0\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") "
Nov 01 06:19:48.866013 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.865991    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-routing-overlay-volume\") pod \"9e374f81-bec1-477b-8ee1-e5631744acb0\" (UID: \"9e374f81-bec1-477b-8ee1-e5631744acb0\") "
Nov 01 06:19:48.866354 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.866335    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-plugins-config-volume" pod "9e374f81-bec1-477b-8ee1-e5631744acb0" (UID: "9e374f81-bec1-477b-8ee1-e5631744acb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:19:48.866434 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.866377    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-routing-overlay-volume" pod "9e374f81-bec1-477b-8ee1-e5631744acb0" (UID: "9e374f81-bec1-477b-8ee1-e5631744acb0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:19:48.866490 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.866462    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:48.868288 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.868266    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-kube-api-access-x92b9" pod "9e374f81-bec1-477b-8ee1-e5631744acb0" (UID: "9e374f81-bec1-477b-8ee1-e5631744acb0"). InnerVolumeSpecName "kube-api-access-x92b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:19:48.868395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.868297    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-provider-credentials" pod "9e374f81-bec1-477b-8ee1-e5631744acb0" (UID: "9e374f81-bec1-477b-8ee1-e5631744acb0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:19:48.967821 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.967789    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:48.967978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.967852    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x92b9\" (UniqueName: \"kubernetes.io/projected/9e374f81-bec1-477b-8ee1-e5631744acb0-kube-api-access-x92b9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:48.967978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:48.967871    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e374f81-bec1-477b-8ee1-e5631744acb0-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:49.825787 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:49.825753    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"
Nov 01 06:19:49.989940 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:49.989910    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"]
Nov 01 06:19:50.039557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:50.039523    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-bfc5mmxpk"]
Nov 01 06:19:51.235281 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:51.235249    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e374f81-bec1-477b-8ee1-e5631744acb0" path="/var/lib/kubelet/pods/9e374f81-bec1-477b-8ee1-e5631744acb0/volumes"
Nov 01 06:19:56.482429 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.482390    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"]
Nov 01 06:19:56.483099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483080    5510 cpu_manager.go:414] "containerMap: removing container" podUID="67a331ec-2379-4453-906a-8cd6aa9996e0" containerName="payload-pre-processing"
Nov 01 06:19:56.483152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483101    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a331ec-2379-4453-906a-8cd6aa9996e0" containerName="payload-pre-processing"
Nov 01 06:19:56.483152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483113    5510 cpu_manager.go:414] "containerMap: removing container" podUID="4108c3c4-065e-4e2a-b9c1-f1d134357b9f" containerName="payload-processing"
Nov 01 06:19:56.483152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483118    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4108c3c4-065e-4e2a-b9c1-f1d134357b9f" containerName="payload-processing"
Nov 01 06:19:56.483152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483129    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9e374f81-bec1-477b-8ee1-e5631744acb0" containerName="payload-processing"
Nov 01 06:19:56.483152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483134    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e374f81-bec1-477b-8ee1-e5631744acb0" containerName="payload-processing"
Nov 01 06:19:56.483152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483145    5510 cpu_manager.go:414] "containerMap: removing container" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy"
Nov 01 06:19:56.483152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483149    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy"
Nov 01 06:19:56.483382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483163    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy"
Nov 01 06:19:56.483382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483171    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy"
Nov 01 06:19:56.483382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483264    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a331ec-2379-4453-906a-8cd6aa9996e0" containerName="payload-pre-processing"
Nov 01 06:19:56.483382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483271    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4108c3c4-065e-4e2a-b9c1-f1d134357b9f" containerName="payload-processing"
Nov 01 06:19:56.483382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483277    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e374f81-bec1-477b-8ee1-e5631744acb0" containerName="payload-processing"
Nov 01 06:19:56.483382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483285    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ba1edb-8e44-4567-8936-6808f39ee230" containerName="istio-proxy"
Nov 01 06:19:56.483382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.483292    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab11984-22df-4438-afa2-a4bed63a76e5" containerName="istio-proxy"
Nov 01 06:19:56.486514 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.486499    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.550759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.550724    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.550960 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.550781    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.550960 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.550801    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.551075 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.550964    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b571d511-15fb-4937-bc74-2c808ab8fec3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.551075 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.551021    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.551185 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.551119    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.551185 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.551153    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.551278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.551225    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.551278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.551257    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpb7\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-kube-api-access-dkpb7\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.652605 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.652571    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.652785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.652634    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.652785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.652658    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.652785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.652695    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b571d511-15fb-4937-bc74-2c808ab8fec3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.652785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.652734    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.652785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.652764    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.653081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.652970    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.653081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.653039    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.653175 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.653095    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpb7\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-kube-api-access-dkpb7\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.653216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.653173    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.653531 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.653511    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.653636 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.653616    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.653698 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.653684    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.653809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.653791    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b571d511-15fb-4937-bc74-2c808ab8fec3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.655692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.655661    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.655877 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.655861    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.713909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.713876    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.714043 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.713952    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpb7\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-kube-api-access-dkpb7\") pod \"maas-default-gateway-openshift-default-855f9574cf-2t54h\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:56.736988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.736926    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"]
Nov 01 06:19:56.800673 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:56.800642    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:57.265865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:57.265820    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"]
Nov 01 06:19:57.268010 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:19:57.267984    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb571d511_15fb_4937_bc74_2c808ab8fec3.slice/crio-4200d1a030a706533df294dbee4f290739cf6b17ee7af232d152f6b9fcc92ab5 WatchSource:0}: Error finding container 4200d1a030a706533df294dbee4f290739cf6b17ee7af232d152f6b9fcc92ab5: Status 404 returned error can't find the container with id 4200d1a030a706533df294dbee4f290739cf6b17ee7af232d152f6b9fcc92ab5
Nov 01 06:19:57.270663 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:57.270623    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:19:57.270742 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:57.270709    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:19:57.270779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:57.270750    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:19:57.878287 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:57.878249    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" event={"ID":"b571d511-15fb-4937-bc74-2c808ab8fec3","Type":"ContainerStarted","Data":"798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7"}
Nov 01 06:19:57.878785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:57.878309    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" event={"ID":"b571d511-15fb-4937-bc74-2c808ab8fec3","Type":"ContainerStarted","Data":"4200d1a030a706533df294dbee4f290739cf6b17ee7af232d152f6b9fcc92ab5"}
Nov 01 06:19:58.049978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:58.049920    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podStartSLOduration=2.049905625 podStartE2EDuration="2.049905625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:58.047205726 +0000 UTC m=+2034.925179138" watchObservedRunningTime="2026-11-01 06:19:58.049905625 +0000 UTC m=+2034.927879034"
Nov 01 06:19:58.801362 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:58.801327    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:19:58.803126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:58.803100    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-2t54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 01 06:19:58.803241 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:58.803150    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 01 06:19:59.801865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:59.801795    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-2t54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 01 06:19:59.802270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:19:59.801892    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 01 06:20:00.801016 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:00.800983    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-2t54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 01 06:20:00.801217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:00.801044    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 01 06:20:01.654879 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:01.654845    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:20:01.801406 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:01.801366    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-2t54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 01 06:20:01.801605 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:01.801438    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 01 06:20:02.809020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:02.808982    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-2t54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 01 06:20:02.809661 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:02.809628    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 01 06:20:03.802036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:03.801996    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-2t54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 01 06:20:03.802230 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:03.802065    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 01 06:20:03.977129 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:03.977097    5510 scope.go:122] "RemoveContainer" containerID="7fe5d34b0b2d31cb67f7ea2b57af92dbf640e42d49affd489986cc6a2f0f550d"
Nov 01 06:20:04.801819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:04.801775    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-2t54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 01 06:20:04.802055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:04.801896    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 01 06:20:06.008733 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:06.008701    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:20:06.009207 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:06.009064    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:20:06.010010 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:06.009989    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:20:06.050910 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:06.050876    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"]
Nov 01 06:20:06.098108 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:06.098064    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a01dd9-qkcrg"]
Nov 01 06:20:07.238587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:07.238556    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8307b4c-2e1c-4b23-b63c-5bab1266ae39" path="/var/lib/kubelet/pods/c8307b4c-2e1c-4b23-b63c-5bab1266ae39/volumes"
Nov 01 06:20:08.577008 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.576979    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"]
Nov 01 06:20:08.582731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.582707    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.590020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.589996    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.590116 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.590026    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.590116 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.590097    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.590195 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.590146    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fxv7\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-kube-api-access-2fxv7\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.635324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.635295    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"kube-root-ca.crt\""
Nov 01 06:20:08.685006 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.684972    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-c481d4f49e7f\""
Nov 01 06:20:08.685187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.684972    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"openshift-service-ca.crt\""
Nov 01 06:20:08.685187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.684990    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8f45a2\""
Nov 01 06:20:08.685187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.684997    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-123858cde139\""
Nov 01 06:20:08.685187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.685027    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-99a8c9586b4b\""
Nov 01 06:20:08.685187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.685028    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-d6d3484085b2\""
Nov 01 06:20:08.685187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.685104    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-e2e--cb1d2723-dockercfg-smk8q\""
Nov 01 06:20:08.687865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.687843    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"]
Nov 01 06:20:08.691009 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.690985    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.691126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.691019    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.691126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.691075    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.691232 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.691156    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxv7\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-kube-api-access-2fxv7\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.691803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.691785    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.693921 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.693903    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.736054 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.736022    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"routing-overlay\""
Nov 01 06:20:08.742140 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.742112    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.755205 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.755172    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxv7\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-kube-api-access-2fxv7\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:08.896154 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:08.896073    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:09.271364 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:09.271331    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"]
Nov 01 06:20:09.272648 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:20:09.272618    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad63f7bd_cc6a_44a2_9065_7c9f905c6c90.slice/crio-eebc34e5b5a26c58841bbda0c37a3fd18c5b769b6c04404c922dfff7edcc5de6 WatchSource:0}: Error finding container eebc34e5b5a26c58841bbda0c37a3fd18c5b769b6c04404c922dfff7edcc5de6: Status 404 returned error can't find the container with id eebc34e5b5a26c58841bbda0c37a3fd18c5b769b6c04404c922dfff7edcc5de6
Nov 01 06:20:09.982836 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:09.982802    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9" event={"ID":"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90","Type":"ContainerStarted","Data":"1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52"}
Nov 01 06:20:09.983275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:09.982903    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9" event={"ID":"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90","Type":"ContainerStarted","Data":"eebc34e5b5a26c58841bbda0c37a3fd18c5b769b6c04404c922dfff7edcc5de6"}
Nov 01 06:20:09.983275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:09.982948    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:10.100397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:10.100338    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9" podStartSLOduration=2.100324108 podStartE2EDuration="2.100324108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:10.099219847 +0000 UTC m=+2046.977193257" watchObservedRunningTime="2026-11-01 06:20:10.100324108 +0000 UTC m=+2046.978297518"
Nov 01 06:20:11.127165 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.127130    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"]
Nov 01 06:20:11.131179 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.131161    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.216745 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.216709    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.216928 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.216903    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nxr\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-kube-api-access-q2nxr\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.216977 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.216964    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.217015 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.216987    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234320    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"openshift-service-ca.crt\""
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234320    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-b2178f695dad\""
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234320    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-bdcae79313a2\""
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234336    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-7fe684b5a48a\""
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234360    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-e2e--a835acb2-dockercfg-kqgqq\""
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234381    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"routing-overlay\""
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234428    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7d25db\""
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234488    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-af808cf4bef1\""
Nov 01 06:20:11.234506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.234498    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"kube-root-ca.crt\""
Nov 01 06:20:11.237394 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.237368    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"]
Nov 01 06:20:11.318113 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.318077    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.318323 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.318118    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.318323 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.318207    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.318822 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.318789    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nxr\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-kube-api-access-q2nxr\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.318953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.318884    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.319082 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.319064    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.320986 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.320970    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.379470 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.379393    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nxr\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-kube-api-access-q2nxr\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.444728 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.444696    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:11.604453 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.604425    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"]
Nov 01 06:20:11.609598 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.609577    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.621692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.621665    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.621794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.621707    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.621794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.621736    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkc8\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-kube-api-access-kwkc8\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.621909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.621811    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66a3206-54e5-419a-a623-ba67e69b3b6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.621909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.621881    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.621993 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.621926    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.621993 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.621952    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.621993 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.621977    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.622096 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.622077    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.722737 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.722702    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.722985 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.722760    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.722985 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.722791    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkc8\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-kube-api-access-kwkc8\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.722985 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.722855    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66a3206-54e5-419a-a623-ba67e69b3b6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.722998    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723046    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723078    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723119    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723407 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723243    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723488    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723506    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723663 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723635    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723859    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.723981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.723883    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66a3206-54e5-419a-a623-ba67e69b3b6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.726350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.726309    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.727022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.726980    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.783962 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.783927    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.784117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.784004    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkc8\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-kube-api-access-kwkc8\") pod \"maas-default-gateway-openshift-default-855f9574cf-xgr5l\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:11.824361 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.824325    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"]
Nov 01 06:20:11.840683 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.840658    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"]
Nov 01 06:20:11.843744 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:20:11.843715    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4483cba7_79fc_4997_b9f6_01b6de27d7fd.slice/crio-7052a56f0b3608d921fba4d05132e83d13ae575597ca73b566a82ad8f3fe3a9d WatchSource:0}: Error finding container 7052a56f0b3608d921fba4d05132e83d13ae575597ca73b566a82ad8f3fe3a9d: Status 404 returned error can't find the container with id 7052a56f0b3608d921fba4d05132e83d13ae575597ca73b566a82ad8f3fe3a9d
Nov 01 06:20:11.923312 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:11.923286    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:12.001916 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:12.001873    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" event={"ID":"4483cba7-79fc-4997-b9f6-01b6de27d7fd","Type":"ContainerStarted","Data":"f0d4e6b1ebe52c1a27f854433c29a9982c385c1267dcfbabe1c22a38e56c59b6"}
Nov 01 06:20:12.002056 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:12.001969    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" event={"ID":"4483cba7-79fc-4997-b9f6-01b6de27d7fd","Type":"ContainerStarted","Data":"7052a56f0b3608d921fba4d05132e83d13ae575597ca73b566a82ad8f3fe3a9d"}
Nov 01 06:20:12.002908 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:12.002846    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:12.119343 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:12.119285    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" podStartSLOduration=1.119266459 podStartE2EDuration="1.119266459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:12.118074661 +0000 UTC m=+2048.996048072" watchObservedRunningTime="2026-11-01 06:20:12.119266459 +0000 UTC m=+2048.997239869"
Nov 01 06:20:12.358621 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:12.358593    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"]
Nov 01 06:20:12.360397 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:20:12.360370    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66a3206_54e5_419a_a623_ba67e69b3b6b.slice/crio-a49ec111685c9f4c9b0b9e3b11476a29ea1716146b88800533f89f5ab2475582 WatchSource:0}: Error finding container a49ec111685c9f4c9b0b9e3b11476a29ea1716146b88800533f89f5ab2475582: Status 404 returned error can't find the container with id a49ec111685c9f4c9b0b9e3b11476a29ea1716146b88800533f89f5ab2475582
Nov 01 06:20:12.363287 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:12.363246    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:20:12.363367 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:12.363349    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:20:12.363411 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:12.363397    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:20:13.009528 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:13.009494    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" event={"ID":"e66a3206-54e5-419a-a623-ba67e69b3b6b","Type":"ContainerStarted","Data":"d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec"}
Nov 01 06:20:13.009702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:13.009556    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" event={"ID":"e66a3206-54e5-419a-a623-ba67e69b3b6b","Type":"ContainerStarted","Data":"a49ec111685c9f4c9b0b9e3b11476a29ea1716146b88800533f89f5ab2475582"}
Nov 01 06:20:13.128751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:13.128690    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podStartSLOduration=2.128670208 podStartE2EDuration="2.128670208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:13.128362427 +0000 UTC m=+2050.006335840" watchObservedRunningTime="2026-11-01 06:20:13.128670208 +0000 UTC m=+2050.006643619"
Nov 01 06:20:13.923730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:13.923695    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:13.925283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:13.925258    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xgr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 01 06:20:13.925397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:13.925307    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 01 06:20:14.923611 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:14.923581    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xgr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 01 06:20:14.923963 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:14.923640    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 01 06:20:15.045629 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:15.045594    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"]
Nov 01 06:20:15.097237 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:15.097205    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2abe5b5-v6qfz"]
Nov 01 06:20:15.237053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:15.237020    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d968108-8c39-46cb-a9a2-ea28fe7d4ffc" path="/var/lib/kubelet/pods/2d968108-8c39-46cb-a9a2-ea28fe7d4ffc/volumes"
Nov 01 06:20:15.923947 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:15.923904    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xgr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 01 06:20:15.924139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:15.923973    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 01 06:20:16.924551 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:16.924518    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xgr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 01 06:20:16.924989 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:16.924572    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 01 06:20:17.546191 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:20:17.546133    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:20:17.924500 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:17.924415    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xgr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 01 06:20:17.924500 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:17.924468    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 01 06:20:18.923950 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:18.923912    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xgr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 01 06:20:18.924128 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:18.923972    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 01 06:20:19.924639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:19.924609    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-xgr5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 01 06:20:19.925078 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:19.924661    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 01 06:20:20.992816 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:20.992787    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:20:21.502428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:21.502400    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:21.502903 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:21.502872    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:21.504546 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:21.504519    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:20:24.019068 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:24.019042    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:20:47.602173 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:20:47.602081    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:20:56.465369 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:20:56.465331    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:21:04.004768 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:04.004729    5510 scope.go:122] "RemoveContainer" containerID="fc49d6c01140c61629c3be2500566cf55a79d3138ceedd7bd8ce2fe6f140a27e"
Nov 01 06:21:11.451988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:11.451959    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"]
Nov 01 06:21:11.452437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:11.452152    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy" containerID="cri-o://a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef" gracePeriod=30
Nov 01 06:21:16.709420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.709395    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:21:16.826307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826229    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-socket\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826454 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826341    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-envoy\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826454 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826377    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-certs\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826454 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826423    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0b5119-4d95-42f1-a501-494364aeee64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0b5119-4d95-42f1-a501-494364aeee64-istiod-ca-cert\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826456    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-kube-api-access-96g4b\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-kube-api-access-96g4b\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826492    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e0b5119-4d95-42f1-a501-494364aeee64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0b5119-4d95-42f1-a501-494364aeee64-istio-podinfo\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826498    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-socket" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:21:16.826613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826574    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-data\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826605    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-istio-token\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-istio-token\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826898 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826677    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-credential-socket\") pod \"0e0b5119-4d95-42f1-a501-494364aeee64\" (UID: \"0e0b5119-4d95-42f1-a501-494364aeee64\") "
Nov 01 06:21:16.826898 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826719    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-certs" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:21:16.826998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826917    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0b5119-4d95-42f1-a501-494364aeee64-istiod-ca-cert" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:21:16.826998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.826959    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-data" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:21:16.827252 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.827172    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-credential-socket" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:21:16.827368 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.827349    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:16.827435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.827377    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:16.827435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.827392    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0b5119-4d95-42f1-a501-494364aeee64-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:16.827435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.827405    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:16.827435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.827419    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:16.828905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.828867    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-envoy" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:21:16.829331 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.829303    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e0b5119-4d95-42f1-a501-494364aeee64-istio-podinfo" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:21:16.829424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.829368    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-kube-api-access-96g4b" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "kube-api-access-96g4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:21:16.829424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.829380    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-istio-token" pod "0e0b5119-4d95-42f1-a501-494364aeee64" (UID: "0e0b5119-4d95-42f1-a501-494364aeee64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:21:16.928792 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.928760    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:16.928792 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.928785    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0b5119-4d95-42f1-a501-494364aeee64-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:16.928792 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.928797    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96g4b\" (UniqueName: \"kubernetes.io/projected/0e0b5119-4d95-42f1-a501-494364aeee64-kube-api-access-96g4b\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:16.929066 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:16.928808    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0b5119-4d95-42f1-a501-494364aeee64-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:17.439267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.439232    5510 generic.go:356] "Generic (PLEG): container finished" podID="0e0b5119-4d95-42f1-a501-494364aeee64" containerID="a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef" exitCode=0
Nov 01 06:21:17.439440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.439284    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"
Nov 01 06:21:17.439440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.439320    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" event={"ID":"0e0b5119-4d95-42f1-a501-494364aeee64","Type":"ContainerDied","Data":"a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef"}
Nov 01 06:21:17.439440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.439378    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq" event={"ID":"0e0b5119-4d95-42f1-a501-494364aeee64","Type":"ContainerDied","Data":"268b8ad74c7f15ba14cf804f1838b21fcb99b63c64a731d5cf8bc450a111ebee"}
Nov 01 06:21:17.439440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.439406    5510 scope.go:122] "RemoveContainer" containerID="a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef"
Nov 01 06:21:17.449344 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.449326    5510 scope.go:122] "RemoveContainer" containerID="a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef"
Nov 01 06:21:17.449567 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:21:17.449548    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef\": container with ID starting with a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef not found: ID does not exist" containerID="a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef"
Nov 01 06:21:17.449634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.449573    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef"} err="failed to get container status \"a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef\": rpc error: code = NotFound desc = could not find container \"a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef\": container with ID starting with a25c6a5ed86f0f592d3485c85a3cd6914a8e2588231b03135c4f3f6a106300ef not found: ID does not exist"
Nov 01 06:21:17.545687 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:21:17.545606    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:21:17.652979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.652948    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"]
Nov 01 06:21:17.702324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:17.702295    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-4s5rq"]
Nov 01 06:21:19.236511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:19.236478    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" path="/var/lib/kubelet/pods/0e0b5119-4d95-42f1-a501-494364aeee64/volumes"
Nov 01 06:21:19.594188 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:19.594102    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:21:47.545708 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:21:47.545672    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:21:58.421975 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:21:58.421940    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:22:04.030755 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:22:04.030722    5510 scope.go:122] "RemoveContainer" containerID="802c064b4d8738459ba90da61f8d83953fb07d9848daab4623c6992978be98af"
Nov 01 06:22:17.546413 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:22:17.546334    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:22:29.352319 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:22:29.352287    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:22:47.545796 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:22:47.545759    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:22:57.246684 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:22:57.246652    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"]
Nov 01 06:22:57.247106 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:22:57.246877    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy" containerID="cri-o://2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c" gracePeriod=30
Nov 01 06:23:02.526149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.526093    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:23:02.576274 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576246    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-podinfo\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.576430 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576330    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-token\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-token\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.576430 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576420    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-certs\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.576555 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576487    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-data\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.576610 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576580    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-credential-socket\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.576665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576614    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f176a81-71c4-47c4-922f-42b022dcd7bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f176a81-71c4-47c4-922f-42b022dcd7bb-istiod-ca-cert\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.576665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576646    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-kube-api-access-75bkj\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-kube-api-access-75bkj\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.576773 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576686    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-socket\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.576848 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.576806    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-envoy\") pod \"2f176a81-71c4-47c4-922f-42b022dcd7bb\" (UID: \"2f176a81-71c4-47c4-922f-42b022dcd7bb\") "
Nov 01 06:23:02.577057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577021    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-data" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.577057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577007    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-credential-socket" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.577230 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577212    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-socket" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.577329 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577303    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-certs" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.577579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577554    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.577856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577580    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.577856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577604    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.577856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577620    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.577856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.577781    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f176a81-71c4-47c4-922f-42b022dcd7bb-istiod-ca-cert" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:23:02.578755 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.578728    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-token" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:02.578938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.578912    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-podinfo" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:23:02.579223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.579198    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-envoy" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.579513 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.579494    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-kube-api-access-75bkj" pod "2f176a81-71c4-47c4-922f-42b022dcd7bb" (UID: "2f176a81-71c4-47c4-922f-42b022dcd7bb"). InnerVolumeSpecName "kube-api-access-75bkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:02.678624 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.678591    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f176a81-71c4-47c4-922f-42b022dcd7bb-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.678624 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.678616    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bkj\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-kube-api-access-75bkj\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.678624 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.678628    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.678931 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.678636    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.678931 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:02.678644    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f176a81-71c4-47c4-922f-42b022dcd7bb-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:03.167849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.164746    5510 generic.go:356] "Generic (PLEG): container finished" podID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerID="2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c" exitCode=0
Nov 01 06:23:03.167849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.164970    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" event={"ID":"2f176a81-71c4-47c4-922f-42b022dcd7bb","Type":"ContainerDied","Data":"2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c"}
Nov 01 06:23:03.167849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.165022    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4" event={"ID":"2f176a81-71c4-47c4-922f-42b022dcd7bb","Type":"ContainerDied","Data":"03d3f23e4d86a400efad4534d920c4b81989f541ffe5cab77da11a7a062c9a74"}
Nov 01 06:23:03.167849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.165048    5510 scope.go:122] "RemoveContainer" containerID="2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c"
Nov 01 06:23:03.167849 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.165219    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"
Nov 01 06:23:03.190681 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.190654    5510 scope.go:122] "RemoveContainer" containerID="2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c"
Nov 01 06:23:03.191069 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:23:03.191041    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c\": container with ID starting with 2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c not found: ID does not exist" containerID="2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c"
Nov 01 06:23:03.191182 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.191077    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c"} err="failed to get container status \"2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c\": rpc error: code = NotFound desc = could not find container \"2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c\": container with ID starting with 2247af855d98f7d5adabac78ab6fd419cd6b427dd557949c6f03217d7853110c not found: ID does not exist"
Nov 01 06:23:03.337490 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.337462    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"]
Nov 01 06:23:03.342271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:03.342244    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-v4bl4"]
Nov 01 06:23:05.236387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:05.236353    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" path="/var/lib/kubelet/pods/2f176a81-71c4-47c4-922f-42b022dcd7bb/volumes"
Nov 01 06:23:17.545687 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:23:17.545653    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:23:39.497188 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:39.497153    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:23:47.546687 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:23:47.546601    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:23:54.732702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.732667    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"]
Nov 01 06:23:54.733268 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.733254    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy"
Nov 01 06:23:54.733327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.733270    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy"
Nov 01 06:23:54.733327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.733296    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy"
Nov 01 06:23:54.733327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.733301    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy"
Nov 01 06:23:54.733424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.733389    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f176a81-71c4-47c4-922f-42b022dcd7bb" containerName="istio-proxy"
Nov 01 06:23:54.733424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.733404    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0b5119-4d95-42f1-a501-494364aeee64" containerName="istio-proxy"
Nov 01 06:23:54.737007 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.736988    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.792077 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.792056    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"kube-root-ca.crt\""
Nov 01 06:23:54.841668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.841641    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3986bf\""
Nov 01 06:23:54.841668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.841657    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-28460a5a0934\""
Nov 01 06:23:54.841936 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.841656    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-d344d6f1c604\""
Nov 01 06:23:54.842004 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.841982    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-3feee30ca48b\""
Nov 01 06:23:54.842004 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.841992    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"routing-overlay\""
Nov 01 06:23:54.842097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.842034    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"openshift-service-ca.crt\""
Nov 01 06:23:54.842148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.842099    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-e2e--cf3b8510-dockercfg-ftjt9\""
Nov 01 06:23:54.844859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.844839    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"]
Nov 01 06:23:54.847618 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.847593    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.847731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.847710    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.847789 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.847770    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.847919 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.847905    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jts\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-kube-api-access-89jts\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.949304 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.949276    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.949449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.949322    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.949449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.949373    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jts\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-kube-api-access-89jts\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.949449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.949421    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.950010 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.949987    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.950148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.950127    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:54.951990 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:54.951972    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:55.009850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:55.009775    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jts\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-kube-api-access-89jts\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:55.050274 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:55.050251    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:55.426524 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:55.426503    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"]
Nov 01 06:23:55.429018 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:23:55.428992    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02d39c1_127e_4f49_817d_19d7ff6ba7a0.slice/crio-535cff7446964a6bafc1ede4f064591d0257d6b25ce7c1466a0b76e3a1d0ddfc WatchSource:0}: Error finding container 535cff7446964a6bafc1ede4f064591d0257d6b25ce7c1466a0b76e3a1d0ddfc: Status 404 returned error can't find the container with id 535cff7446964a6bafc1ede4f064591d0257d6b25ce7c1466a0b76e3a1d0ddfc
Nov 01 06:23:55.482870 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:55.482818    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" event={"ID":"e02d39c1-127e-4f49-817d-19d7ff6ba7a0","Type":"ContainerStarted","Data":"535cff7446964a6bafc1ede4f064591d0257d6b25ce7c1466a0b76e3a1d0ddfc"}
Nov 01 06:23:56.520688 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:56.519236    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" event={"ID":"e02d39c1-127e-4f49-817d-19d7ff6ba7a0","Type":"ContainerStarted","Data":"daab718057c84b23593d71c7d8c2e682976081aedf28b4ad83045eb8a10b088b"}
Nov 01 06:23:56.520688 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:56.520220    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:23:56.637157 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:23:56.637100    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podStartSLOduration=2.6370865869999998 podStartE2EDuration="2.637086587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:56.635446705 +0000 UTC m=+2273.513420115" watchObservedRunningTime="2026-11-01 06:23:56.637086587 +0000 UTC m=+2273.515059997"
Nov 01 06:24:08.534086 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:08.534052    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:24:17.545409 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:24:17.545375    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:24:18.532527 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:18.532485    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:24:25.001400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.001368    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"]
Nov 01 06:24:25.005902 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.005886    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.060603 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.060499    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-83558e40-serving-cert\""
Nov 01 06:24:25.112552 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.112516    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"]
Nov 01 06:24:25.139566 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.139535    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b67cf-03c1-4f78-980b-d72ef41e09cc-openshift-service-ca\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.139566 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.139568    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-metrics-certs\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.139760 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.139607    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.139866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.139757    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlr8\" (UniqueName: \"kubernetes.io/projected/125b67cf-03c1-4f78-980b-d72ef41e09cc-kube-api-access-kwlr8\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.241409 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.241380    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlr8\" (UniqueName: \"kubernetes.io/projected/125b67cf-03c1-4f78-980b-d72ef41e09cc-kube-api-access-kwlr8\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.241595 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.241515    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b67cf-03c1-4f78-980b-d72ef41e09cc-openshift-service-ca\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.241595 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.241541    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-metrics-certs\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.241595 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.241570    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.241760 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:24:25.241747    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-serving-cert: secret "maas-api-e2e-ait-83558e40-serving-cert" not found
Nov 01 06:24:25.241811 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:24:25.241802    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls podName:125b67cf-03c1-4f78-980b-d72ef41e09cc nodeName:}" failed. No retries permitted until 2026-11-01 06:24:25.741788066 +0000 UTC m=+2302.619761466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls") pod "maas-api-e2e-ait-83558e40-68c64784b5-qxhtq" (UID: "125b67cf-03c1-4f78-980b-d72ef41e09cc") : secret "maas-api-e2e-ait-83558e40-serving-cert" not found
Nov 01 06:24:25.242279 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.242260    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b67cf-03c1-4f78-980b-d72ef41e09cc-openshift-service-ca\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.244163 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.244137    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-metrics-certs\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.325901 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.315698    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlr8\" (UniqueName: \"kubernetes.io/projected/125b67cf-03c1-4f78-980b-d72ef41e09cc-kube-api-access-kwlr8\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.679005 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.678918    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:24:25.748516 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.748486    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.751044 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.751026    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls\") pod \"maas-api-e2e-ait-83558e40-68c64784b5-qxhtq\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:25.922292 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:25.920054    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:26.324448 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:26.324425    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"]
Nov 01 06:24:26.326238 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:24:26.326201    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125b67cf_03c1_4f78_980b_d72ef41e09cc.slice/crio-e26ea99a6db1cce1ebfdb4edc92c3e67faca1cd0e31b54968a4e53859d018136 WatchSource:0}: Error finding container e26ea99a6db1cce1ebfdb4edc92c3e67faca1cd0e31b54968a4e53859d018136: Status 404 returned error can't find the container with id e26ea99a6db1cce1ebfdb4edc92c3e67faca1cd0e31b54968a4e53859d018136
Nov 01 06:24:26.327541 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:26.327524    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:24:26.701661 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:26.701590    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq" event={"ID":"125b67cf-03c1-4f78-980b-d72ef41e09cc","Type":"ContainerStarted","Data":"e26ea99a6db1cce1ebfdb4edc92c3e67faca1cd0e31b54968a4e53859d018136"}
Nov 01 06:24:27.709317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:27.709276    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq" event={"ID":"125b67cf-03c1-4f78-980b-d72ef41e09cc","Type":"ContainerStarted","Data":"7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855"}
Nov 01 06:24:27.709724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:27.709343    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:27.836794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:27.836735    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq" podStartSLOduration=3.031052086 podStartE2EDuration="3.836721358s" totalImagesPullingTime="805.669272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:27.835154494 +0000 UTC m=+2304.713127905" watchObservedRunningTime="2026-11-01 06:24:27.836721358 +0000 UTC m=+2304.714694768"
Nov 01 06:24:28.532680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:28.532637    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:24:33.722285 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:33.722257    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:24:38.532319 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:38.532282    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:24:47.545918 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:24:47.545880    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:24:48.532688 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:48.532651    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:24:58.532653 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:24:58.532611    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:02.229610 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:02.229564    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:09.630107 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:09.630068    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:25:12.229886 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:12.229842    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:17.545074 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:17.544979    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:25:22.237006 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:22.236967    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:22.841117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:22.841080    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"]
Nov 01 06:25:22.842469 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:22.842440    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:27.841610 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:27.841553    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing" containerID="cri-o://daab718057c84b23593d71c7d8c2e682976081aedf28b4ad83045eb8a10b088b" gracePeriod=25
Nov 01 06:25:28.109018 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.108988    5510 generic.go:356] "Generic (PLEG): container finished" podID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerID="daab718057c84b23593d71c7d8c2e682976081aedf28b4ad83045eb8a10b088b" exitCode=0
Nov 01 06:25:28.109160 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.109057    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" event={"ID":"e02d39c1-127e-4f49-817d-19d7ff6ba7a0","Type":"ContainerDied","Data":"daab718057c84b23593d71c7d8c2e682976081aedf28b4ad83045eb8a10b088b"}
Nov 01 06:25:28.109160 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.109108    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv" event={"ID":"e02d39c1-127e-4f49-817d-19d7ff6ba7a0","Type":"ContainerDied","Data":"535cff7446964a6bafc1ede4f064591d0257d6b25ce7c1466a0b76e3a1d0ddfc"}
Nov 01 06:25:28.109160 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.109131    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="535cff7446964a6bafc1ede4f064591d0257d6b25ce7c1466a0b76e3a1d0ddfc"
Nov 01 06:25:28.112343 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.112324    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:25:28.156880 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.156851    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-routing-overlay-volume\") pod \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") "
Nov 01 06:25:28.157072 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.156901    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-plugins-config-volume\") pod \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") "
Nov 01 06:25:28.157072 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.156938    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-provider-credentials\") pod \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") "
Nov 01 06:25:28.157200 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.157143    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-kube-api-access-89jts\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-kube-api-access-89jts\") pod \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\" (UID: \"e02d39c1-127e-4f49-817d-19d7ff6ba7a0\") "
Nov 01 06:25:28.157308 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.157288    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-routing-overlay-volume" pod "e02d39c1-127e-4f49-817d-19d7ff6ba7a0" (UID: "e02d39c1-127e-4f49-817d-19d7ff6ba7a0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:28.157367 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.157326    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-plugins-config-volume" pod "e02d39c1-127e-4f49-817d-19d7ff6ba7a0" (UID: "e02d39c1-127e-4f49-817d-19d7ff6ba7a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:28.157514 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.157486    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:28.157514 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.157512    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:28.159297 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.159274    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-provider-credentials" pod "e02d39c1-127e-4f49-817d-19d7ff6ba7a0" (UID: "e02d39c1-127e-4f49-817d-19d7ff6ba7a0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:28.159401 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.159310    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-kube-api-access-89jts" pod "e02d39c1-127e-4f49-817d-19d7ff6ba7a0" (UID: "e02d39c1-127e-4f49-817d-19d7ff6ba7a0"). InnerVolumeSpecName "kube-api-access-89jts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:28.258850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.258797    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:28.259120 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:28.259089    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89jts\" (UniqueName: \"kubernetes.io/projected/e02d39c1-127e-4f49-817d-19d7ff6ba7a0-kube-api-access-89jts\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:29.114977 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:29.114948    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"
Nov 01 06:25:29.278400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:29.278370    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"]
Nov 01 06:25:29.327998 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:29.327968    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77drfsv"]
Nov 01 06:25:31.236134 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:31.236092    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" path="/var/lib/kubelet/pods/e02d39c1-127e-4f49-817d-19d7ff6ba7a0/volumes"
Nov 01 06:25:35.140927 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.140892    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"]
Nov 01 06:25:35.141771 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.141750    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing"
Nov 01 06:25:35.141904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.141772    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing"
Nov 01 06:25:35.141999 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.141984    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02d39c1-127e-4f49-817d-19d7ff6ba7a0" containerName="payload-processing"
Nov 01 06:25:35.146620 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.146602    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.202488 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.202452    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"]
Nov 01 06:25:35.279670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.279627    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.280309 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.280286    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.280595 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.280574    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.282286 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.282265    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnsbp\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-kube-api-access-lnsbp\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.383515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.383476    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.383727 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.383554    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.383813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.383780    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnsbp\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-kube-api-access-lnsbp\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.383930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.383899    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.384482 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.384459    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.384757 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.384732    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.386665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.386638    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.444339 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.444270    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnsbp\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-kube-api-access-lnsbp\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.460844 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.460807    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:35.835818 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:35.835791    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"]
Nov 01 06:25:35.837712 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:25:35.837688    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb971019_8312_404c_92be_096040a93c78.slice/crio-a0ac4580d2abd158c246004c8af7d3b383fe7943dd275681215912e6904fe4b4 WatchSource:0}: Error finding container a0ac4580d2abd158c246004c8af7d3b383fe7943dd275681215912e6904fe4b4: Status 404 returned error can't find the container with id a0ac4580d2abd158c246004c8af7d3b383fe7943dd275681215912e6904fe4b4
Nov 01 06:25:36.158761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:36.158673    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" event={"ID":"bb971019-8312-404c-92be-096040a93c78","Type":"ContainerStarted","Data":"c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24"}
Nov 01 06:25:36.158761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:36.158736    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" event={"ID":"bb971019-8312-404c-92be-096040a93c78","Type":"ContainerStarted","Data":"a0ac4580d2abd158c246004c8af7d3b383fe7943dd275681215912e6904fe4b4"}
Nov 01 06:25:36.159219 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:36.158764    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:36.274850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:36.274785    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" podStartSLOduration=1.2747666020000001 podStartE2EDuration="1.274766602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:36.273461991 +0000 UTC m=+2373.151435413" watchObservedRunningTime="2026-11-01 06:25:36.274766602 +0000 UTC m=+2373.152740013"
Nov 01 06:25:37.316262 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:37.316230    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-123858cde139: secret "agc-cred-123858cde139" not found
Nov 01 06:25:37.316262 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:37.316269    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc: secret "agc-cred-123858cde139" not found
Nov 01 06:25:37.316743 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:37.316330    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials podName:bb971019-8312-404c-92be-096040a93c78 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:37.816309912 +0000 UTC m=+2374.694283317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" (UID: "bb971019-8312-404c-92be-096040a93c78") : secret "agc-cred-123858cde139" not found
Nov 01 06:25:37.822770 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:37.822735    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-123858cde139: secret "agc-cred-123858cde139" not found
Nov 01 06:25:37.822770 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:37.822764    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc: secret "agc-cred-123858cde139" not found
Nov 01 06:25:37.823016 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:37.822817    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials podName:bb971019-8312-404c-92be-096040a93c78 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:38.822801723 +0000 UTC m=+2375.700775129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" (UID: "bb971019-8312-404c-92be-096040a93c78") : secret "agc-cred-123858cde139" not found
Nov 01 06:25:38.852740 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:38.852695    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-123858cde139: secret "agc-cred-123858cde139" not found
Nov 01 06:25:38.852740 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:38.852736    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc: secret "agc-cred-123858cde139" not found
Nov 01 06:25:38.853188 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:38.852794    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials podName:bb971019-8312-404c-92be-096040a93c78 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:40.852779448 +0000 UTC m=+2377.730752853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" (UID: "bb971019-8312-404c-92be-096040a93c78") : secret "agc-cred-123858cde139" not found
Nov 01 06:25:40.311706 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.311664    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"]
Nov 01 06:25:40.314774 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.314745    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" podUID="bb971019-8312-404c-92be-096040a93c78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:40.344213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.344179    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"]
Nov 01 06:25:40.349072 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.349052    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.372754 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.372726    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.372982 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.372957    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.373039 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.372994    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.373039 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.373020    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdfp\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-kube-api-access-7gdfp\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.423086 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.423053    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"]
Nov 01 06:25:40.474053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.474012    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.474053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.474059    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdfp\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-kube-api-access-7gdfp\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.474321 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.474136    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.474381 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.474343    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.474724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.474696    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.475061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.475034    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.477371 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.477343    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.554077 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.554045    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdfp\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-kube-api-access-7gdfp\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.664471 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.664376    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:40.867437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:40.867403    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:25:40.879748 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:40.879713    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-123858cde139: secret "agc-cred-123858cde139" not found
Nov 01 06:25:40.879748 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:40.879752    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc: secret "agc-cred-123858cde139" not found
Nov 01 06:25:40.879987 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:40.879823    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials podName:bb971019-8312-404c-92be-096040a93c78 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:44.879803859 +0000 UTC m=+2381.757777248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" (UID: "bb971019-8312-404c-92be-096040a93c78") : secret "agc-cred-123858cde139" not found
Nov 01 06:25:41.042012 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:41.041983    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"]
Nov 01 06:25:41.043380 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:25:41.043357    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b8d892_6a42_448d_b36f_47735891c922.slice/crio-6ae331710e638f392e19208986b0278a25319f179b0cedf919e5e3b8641a396e WatchSource:0}: Error finding container 6ae331710e638f392e19208986b0278a25319f179b0cedf919e5e3b8641a396e: Status 404 returned error can't find the container with id 6ae331710e638f392e19208986b0278a25319f179b0cedf919e5e3b8641a396e
Nov 01 06:25:41.217032 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:41.216994    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" event={"ID":"d2b8d892-6a42-448d-b36f-47735891c922","Type":"ContainerStarted","Data":"d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2"}
Nov 01 06:25:41.217228 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:41.217057    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:25:41.217228 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:41.217079    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" event={"ID":"d2b8d892-6a42-448d-b36f-47735891c922","Type":"ContainerStarted","Data":"6ae331710e638f392e19208986b0278a25319f179b0cedf919e5e3b8641a396e"}
Nov 01 06:25:41.333309 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:41.333205    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" podStartSLOduration=1.3331876870000001 podStartE2EDuration="1.333187687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:41.331986816 +0000 UTC m=+2378.209960227" watchObservedRunningTime="2026-11-01 06:25:41.333187687 +0000 UTC m=+2378.211161133"
Nov 01 06:25:44.942172 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:44.942138    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-123858cde139: secret "agc-cred-123858cde139" not found
Nov 01 06:25:44.942172 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:44.942169    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc: secret "agc-cred-123858cde139" not found
Nov 01 06:25:44.942638 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:44.942222    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials podName:bb971019-8312-404c-92be-096040a93c78 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:52.94220763 +0000 UTC m=+2389.820181019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" (UID: "bb971019-8312-404c-92be-096040a93c78") : secret "agc-cred-123858cde139" not found
Nov 01 06:25:45.312463 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.312424    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" podUID="bb971019-8312-404c-92be-096040a93c78" containerName="payload-processing" containerID="cri-o://c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24" gracePeriod=25
Nov 01 06:25:45.583601 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.583577    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:45.648241 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.648213    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-kube-api-access-lnsbp\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-kube-api-access-lnsbp\") pod \"bb971019-8312-404c-92be-096040a93c78\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") "
Nov 01 06:25:45.648395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.648315    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-plugins-config-volume\") pod \"bb971019-8312-404c-92be-096040a93c78\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") "
Nov 01 06:25:45.648437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.648417    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials\") pod \"bb971019-8312-404c-92be-096040a93c78\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") "
Nov 01 06:25:45.648565 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.648549    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-routing-overlay-volume\") pod \"bb971019-8312-404c-92be-096040a93c78\" (UID: \"bb971019-8312-404c-92be-096040a93c78\") "
Nov 01 06:25:45.648700 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.648671    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-plugins-config-volume" pod "bb971019-8312-404c-92be-096040a93c78" (UID: "bb971019-8312-404c-92be-096040a93c78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:45.648969 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.648940    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-routing-overlay-volume" pod "bb971019-8312-404c-92be-096040a93c78" (UID: "bb971019-8312-404c-92be-096040a93c78"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:45.649225 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.649020    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:45.650606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.650582    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-kube-api-access-lnsbp" pod "bb971019-8312-404c-92be-096040a93c78" (UID: "bb971019-8312-404c-92be-096040a93c78"). InnerVolumeSpecName "kube-api-access-lnsbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:45.650682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.650627    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials" pod "bb971019-8312-404c-92be-096040a93c78" (UID: "bb971019-8312-404c-92be-096040a93c78"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:45.750411 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.750381    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:45.750411 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.750409    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb971019-8312-404c-92be-096040a93c78-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:45.750601 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:45.750419    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnsbp\" (UniqueName: \"kubernetes.io/projected/bb971019-8312-404c-92be-096040a93c78-kube-api-access-lnsbp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:46.257221 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.257189    5510 generic.go:356] "Generic (PLEG): container finished" podID="bb971019-8312-404c-92be-096040a93c78" containerID="c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24" exitCode=0
Nov 01 06:25:46.257632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.257232    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"
Nov 01 06:25:46.257632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.257275    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" event={"ID":"bb971019-8312-404c-92be-096040a93c78","Type":"ContainerDied","Data":"c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24"}
Nov 01 06:25:46.257632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.257339    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc" event={"ID":"bb971019-8312-404c-92be-096040a93c78","Type":"ContainerDied","Data":"a0ac4580d2abd158c246004c8af7d3b383fe7943dd275681215912e6904fe4b4"}
Nov 01 06:25:46.257632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.257362    5510 scope.go:122] "RemoveContainer" containerID="c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24"
Nov 01 06:25:46.271087 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.271066    5510 scope.go:122] "RemoveContainer" containerID="c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24"
Nov 01 06:25:46.271380 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:46.271357    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24\": container with ID starting with c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24 not found: ID does not exist" containerID="c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24"
Nov 01 06:25:46.271474 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.271385    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24"} err="failed to get container status \"c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24\": rpc error: code = NotFound desc = could not find container \"c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24\": container with ID starting with c85ca5ca2eb972f326333382c00a2fdd340215053ec9ddf45d17708f9708aa24 not found: ID does not exist"
Nov 01 06:25:46.421253 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.421222    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"]
Nov 01 06:25:46.470561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:46.470531    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9hcmrc"]
Nov 01 06:25:47.244033 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:47.243997    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb971019-8312-404c-92be-096040a93c78" path="/var/lib/kubelet/pods/bb971019-8312-404c-92be-096040a93c78/volumes"
Nov 01 06:25:47.545805 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:25:47.545720    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:25:52.225589 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:25:52.225545    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" podUID="d2b8d892-6a42-448d-b36f-47735891c922" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:02.225180 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:02.225138    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" podUID="d2b8d892-6a42-448d-b36f-47735891c922" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:12.224957 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:12.224912    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" podUID="d2b8d892-6a42-448d-b36f-47735891c922" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:15.185423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.185386    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"]
Nov 01 06:26:15.186550 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.186475    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bb971019-8312-404c-92be-096040a93c78" containerName="payload-processing"
Nov 01 06:26:15.186550 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.186497    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb971019-8312-404c-92be-096040a93c78" containerName="payload-processing"
Nov 01 06:26:15.186694 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.186620    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb971019-8312-404c-92be-096040a93c78" containerName="payload-processing"
Nov 01 06:26:15.191395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.191377    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.295988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.295955    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"]
Nov 01 06:26:15.317752 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.317717    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.317991 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.317968    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbsw\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-kube-api-access-gmbsw\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.318113 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.318093    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.318169 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.318127    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.419820 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.419784    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbsw\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-kube-api-access-gmbsw\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.420020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.419897    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.420020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.419923    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.420020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.419946    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.420673 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.420650    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.420812 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.420729    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.422506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.422488    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.481016 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.480986    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbsw\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-kube-api-access-gmbsw\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.505719 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.505697    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:15.888550 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:15.888522    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"]
Nov 01 06:26:15.890412 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:26:15.890384    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2b5837_29fb_42dc_a907_6ceb361ad78e.slice/crio-3e317829994be431a2e3525b531936e3be6dcb9674c0596e01e497da882cb646 WatchSource:0}: Error finding container 3e317829994be431a2e3525b531936e3be6dcb9674c0596e01e497da882cb646: Status 404 returned error can't find the container with id 3e317829994be431a2e3525b531936e3be6dcb9674c0596e01e497da882cb646
Nov 01 06:26:16.548353 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:16.548311    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" event={"ID":"fa2b5837-29fb-42dc-a907-6ceb361ad78e","Type":"ContainerStarted","Data":"08fb7179f23fc9c1bf44db048c314360ba469ba482cb53aa6a85a8ff24fa9f5a"}
Nov 01 06:26:16.548732 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:16.548385    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:16.548732 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:16.548410    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" event={"ID":"fa2b5837-29fb-42dc-a907-6ceb361ad78e","Type":"ContainerStarted","Data":"3e317829994be431a2e3525b531936e3be6dcb9674c0596e01e497da882cb646"}
Nov 01 06:26:16.665573 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:16.665519    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" podStartSLOduration=1.665505362 podStartE2EDuration="1.665505362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:16.664440695 +0000 UTC m=+2413.542414131" watchObservedRunningTime="2026-11-01 06:26:16.665505362 +0000 UTC m=+2413.543478772"
Nov 01 06:26:17.545801 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:17.545762    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:26:17.703610 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:17.703578    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"]
Nov 01 06:26:17.704974 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:17.704935    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" podUID="d2b8d892-6a42-448d-b36f-47735891c922" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:22.704450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:22.704408    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" podUID="d2b8d892-6a42-448d-b36f-47735891c922" containerName="payload-processing" containerID="cri-o://d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2" gracePeriod=25
Nov 01 06:26:22.956288 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:22.956226    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:26:23.001927 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.001899    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-provider-credentials\") pod \"d2b8d892-6a42-448d-b36f-47735891c922\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") "
Nov 01 06:26:23.002076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.001932    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-kube-api-access-7gdfp\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-kube-api-access-7gdfp\") pod \"d2b8d892-6a42-448d-b36f-47735891c922\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") "
Nov 01 06:26:23.002147 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.002079    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-routing-overlay-volume\") pod \"d2b8d892-6a42-448d-b36f-47735891c922\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") "
Nov 01 06:26:23.002147 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.002122    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-plugins-config-volume\") pod \"d2b8d892-6a42-448d-b36f-47735891c922\" (UID: \"d2b8d892-6a42-448d-b36f-47735891c922\") "
Nov 01 06:26:23.002475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.002430    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-routing-overlay-volume" pod "d2b8d892-6a42-448d-b36f-47735891c922" (UID: "d2b8d892-6a42-448d-b36f-47735891c922"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:23.002605 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.002549    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-plugins-config-volume" pod "d2b8d892-6a42-448d-b36f-47735891c922" (UID: "d2b8d892-6a42-448d-b36f-47735891c922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:23.002667 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.002631    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:23.004273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.004252    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-provider-credentials" pod "d2b8d892-6a42-448d-b36f-47735891c922" (UID: "d2b8d892-6a42-448d-b36f-47735891c922"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:23.004273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.004260    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-kube-api-access-7gdfp" pod "d2b8d892-6a42-448d-b36f-47735891c922" (UID: "d2b8d892-6a42-448d-b36f-47735891c922"). InnerVolumeSpecName "kube-api-access-7gdfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:23.103401 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.103368    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:23.103401 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.103401    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gdfp\" (UniqueName: \"kubernetes.io/projected/d2b8d892-6a42-448d-b36f-47735891c922-kube-api-access-7gdfp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:23.103669 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.103411    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b8d892-6a42-448d-b36f-47735891c922-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:23.306047 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.305983    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-7fe684b5a48a: secret "agc-cred-7fe684b5a48a" not found
Nov 01 06:26:23.306047 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.306010    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-af808cf4bef1: secret "agc-cred-af808cf4bef1" not found
Nov 01 06:26:23.306047 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.306019    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-b2178f695dad: secret "agc-cred-b2178f695dad" not found
Nov 01 06:26:23.306047 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.306036    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6: [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:23.306369 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.306099    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials podName:4483cba7-79fc-4997-b9f6-01b6de27d7fd nodeName:}" failed. No retries permitted until 2026-11-01 06:26:23.806082507 +0000 UTC m=+2420.684055909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd") : [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:23.593089 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.593001    5510 generic.go:356] "Generic (PLEG): container finished" podID="d2b8d892-6a42-448d-b36f-47735891c922" containerID="d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2" exitCode=0
Nov 01 06:26:23.593089 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.593055    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"
Nov 01 06:26:23.593283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.593056    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" event={"ID":"d2b8d892-6a42-448d-b36f-47735891c922","Type":"ContainerDied","Data":"d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2"}
Nov 01 06:26:23.593283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.593185    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9" event={"ID":"d2b8d892-6a42-448d-b36f-47735891c922","Type":"ContainerDied","Data":"6ae331710e638f392e19208986b0278a25319f179b0cedf919e5e3b8641a396e"}
Nov 01 06:26:23.593283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.593216    5510 scope.go:122] "RemoveContainer" containerID="d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2"
Nov 01 06:26:23.603101 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.603083    5510 scope.go:122] "RemoveContainer" containerID="d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2"
Nov 01 06:26:23.603378 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.603359    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2\": container with ID starting with d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2 not found: ID does not exist" containerID="d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2"
Nov 01 06:26:23.603424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.603389    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2"} err="failed to get container status \"d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2\": rpc error: code = NotFound desc = could not find container \"d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2\": container with ID starting with d74e6f9624bbfd88cf8309cc562f3fdf701f135c75f04a24f5429a79953ed5b2 not found: ID does not exist"
Nov 01 06:26:23.767348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.767313    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"]
Nov 01 06:26:23.813301 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.813269    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-7fe684b5a48a: secret "agc-cred-7fe684b5a48a" not found
Nov 01 06:26:23.813301 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.813296    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-af808cf4bef1: secret "agc-cred-af808cf4bef1" not found
Nov 01 06:26:23.813301 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.813309    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-b2178f695dad: secret "agc-cred-b2178f695dad" not found
Nov 01 06:26:23.813599 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.813326    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6: [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:23.813599 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:23.813390    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials podName:4483cba7-79fc-4997-b9f6-01b6de27d7fd nodeName:}" failed. No retries permitted until 2026-11-01 06:26:24.813371987 +0000 UTC m=+2421.691345374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd") : [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:23.815393 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:23.815371    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-696dg7zk9"]
Nov 01 06:26:24.029856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:24.028986    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:26:24.837402 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:24.837368    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-7fe684b5a48a: secret "agc-cred-7fe684b5a48a" not found
Nov 01 06:26:24.837402 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:24.837399    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-af808cf4bef1: secret "agc-cred-af808cf4bef1" not found
Nov 01 06:26:24.837402 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:24.837409    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-b2178f695dad: secret "agc-cred-b2178f695dad" not found
Nov 01 06:26:24.838074 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:24.837425    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6: [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:24.838074 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:24.837492    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials podName:4483cba7-79fc-4997-b9f6-01b6de27d7fd nodeName:}" failed. No retries permitted until 2026-11-01 06:26:26.837472162 +0000 UTC m=+2423.715445565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd") : [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:25.238754 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:25.238718    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b8d892-6a42-448d-b36f-47735891c922" path="/var/lib/kubelet/pods/d2b8d892-6a42-448d-b36f-47735891c922/volumes"
Nov 01 06:26:25.754388 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:25.754360    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"]
Nov 01 06:26:25.754579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:25.754542    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq" podUID="125b67cf-03c1-4f78-980b-d72ef41e09cc" containerName="maas-api" containerID="cri-o://7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855" gracePeriod=30
Nov 01 06:26:26.448448 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.448421    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:26:26.542254 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.542203    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125b67cf-03c1-4f78-980b-d72ef41e09cc-kube-api-access-kwlr8\" (UniqueName: \"kubernetes.io/projected/125b67cf-03c1-4f78-980b-d72ef41e09cc-kube-api-access-kwlr8\") pod \"125b67cf-03c1-4f78-980b-d72ef41e09cc\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") "
Nov 01 06:26:26.542442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.542279    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-metrics-certs\") pod \"125b67cf-03c1-4f78-980b-d72ef41e09cc\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") "
Nov 01 06:26:26.542442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.542427    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125b67cf-03c1-4f78-980b-d72ef41e09cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b67cf-03c1-4f78-980b-d72ef41e09cc-openshift-service-ca\") pod \"125b67cf-03c1-4f78-980b-d72ef41e09cc\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") "
Nov 01 06:26:26.542532 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.542519    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls\") pod \"125b67cf-03c1-4f78-980b-d72ef41e09cc\" (UID: \"125b67cf-03c1-4f78-980b-d72ef41e09cc\") "
Nov 01 06:26:26.543158 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.543125    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125b67cf-03c1-4f78-980b-d72ef41e09cc-openshift-service-ca" pod "125b67cf-03c1-4f78-980b-d72ef41e09cc" (UID: "125b67cf-03c1-4f78-980b-d72ef41e09cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:26.549261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.549220    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-metrics-certs" pod "125b67cf-03c1-4f78-980b-d72ef41e09cc" (UID: "125b67cf-03c1-4f78-980b-d72ef41e09cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:26.549354 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.549292    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125b67cf-03c1-4f78-980b-d72ef41e09cc-kube-api-access-kwlr8" pod "125b67cf-03c1-4f78-980b-d72ef41e09cc" (UID: "125b67cf-03c1-4f78-980b-d72ef41e09cc"). InnerVolumeSpecName "kube-api-access-kwlr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:26.549518 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.549501    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls" pod "125b67cf-03c1-4f78-980b-d72ef41e09cc" (UID: "125b67cf-03c1-4f78-980b-d72ef41e09cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:26.651953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.650606    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b67cf-03c1-4f78-980b-d72ef41e09cc-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:26.651953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.650641    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:26.651953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.650656    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwlr8\" (UniqueName: \"kubernetes.io/projected/125b67cf-03c1-4f78-980b-d72ef41e09cc-kube-api-access-kwlr8\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:26.651953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.650669    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125b67cf-03c1-4f78-980b-d72ef41e09cc-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:26.749032 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.748993    5510 generic.go:356] "Generic (PLEG): container finished" podID="125b67cf-03c1-4f78-980b-d72ef41e09cc" containerID="7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855" exitCode=0
Nov 01 06:26:26.749273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.749249    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq" event={"ID":"125b67cf-03c1-4f78-980b-d72ef41e09cc","Type":"ContainerDied","Data":"7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855"}
Nov 01 06:26:26.749431 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.749414    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq" event={"ID":"125b67cf-03c1-4f78-980b-d72ef41e09cc","Type":"ContainerDied","Data":"e26ea99a6db1cce1ebfdb4edc92c3e67faca1cd0e31b54968a4e53859d018136"}
Nov 01 06:26:26.749548 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.749535    5510 scope.go:122] "RemoveContainer" containerID="7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855"
Nov 01 06:26:26.749807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.749788    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"
Nov 01 06:26:26.834731 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.834197    5510 scope.go:122] "RemoveContainer" containerID="7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855"
Nov 01 06:26:26.835116 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:26.835012    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855\": container with ID starting with 7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855 not found: ID does not exist" containerID="7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855"
Nov 01 06:26:26.835116 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.835051    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855"} err="failed to get container status \"7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855\": rpc error: code = NotFound desc = could not find container \"7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855\": container with ID starting with 7c8c037b71a4715b7c753cb724e9b88122b2ff05f85f1628b581d69e107b6855 not found: ID does not exist"
Nov 01 06:26:26.856129 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:26.854913    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-7fe684b5a48a: secret "agc-cred-7fe684b5a48a" not found
Nov 01 06:26:26.856129 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:26.854946    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-af808cf4bef1: secret "agc-cred-af808cf4bef1" not found
Nov 01 06:26:26.856129 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:26.854957    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-b2178f695dad: secret "agc-cred-b2178f695dad" not found
Nov 01 06:26:26.856129 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:26.854974    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6: [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:26.856129 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:26.855038    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials podName:4483cba7-79fc-4997-b9f6-01b6de27d7fd nodeName:}" failed. No retries permitted until 2026-11-01 06:26:30.855018942 +0000 UTC m=+2427.732992344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd") : [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:26.951910 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.951455    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"]
Nov 01 06:26:26.954213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:26.954189    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e40-68c64784b5-qxhtq"]
Nov 01 06:26:27.247164 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:27.247121    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125b67cf-03c1-4f78-980b-d72ef41e09cc" path="/var/lib/kubelet/pods/125b67cf-03c1-4f78-980b-d72ef41e09cc/volumes"
Nov 01 06:26:27.558621 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:27.558523    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" podUID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:30.918377 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:30.918343    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-7fe684b5a48a: secret "agc-cred-7fe684b5a48a" not found
Nov 01 06:26:30.918377 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:30.918371    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-af808cf4bef1: secret "agc-cred-af808cf4bef1" not found
Nov 01 06:26:30.918377 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:30.918378    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-b2178f695dad: secret "agc-cred-b2178f695dad" not found
Nov 01 06:26:30.918899 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:30.918391    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6: [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:30.918899 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:30.918464    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials podName:4483cba7-79fc-4997-b9f6-01b6de27d7fd nodeName:}" failed. No retries permitted until 2026-11-01 06:26:38.918448356 +0000 UTC m=+2435.796421744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd") : [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found]
Nov 01 06:26:37.556086 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:37.556040    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" podUID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:38.168515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.168478    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"]
Nov 01 06:26:38.169206 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.169183    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d2b8d892-6a42-448d-b36f-47735891c922" containerName="payload-processing"
Nov 01 06:26:38.169206 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.169202    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b8d892-6a42-448d-b36f-47735891c922" containerName="payload-processing"
Nov 01 06:26:38.169370 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.169240    5510 cpu_manager.go:414] "containerMap: removing container" podUID="125b67cf-03c1-4f78-980b-d72ef41e09cc" containerName="maas-api"
Nov 01 06:26:38.169370 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.169245    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125b67cf-03c1-4f78-980b-d72ef41e09cc" containerName="maas-api"
Nov 01 06:26:38.169370 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.169341    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="125b67cf-03c1-4f78-980b-d72ef41e09cc" containerName="maas-api"
Nov 01 06:26:38.169370 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.169355    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b8d892-6a42-448d-b36f-47735891c922" containerName="payload-processing"
Nov 01 06:26:38.173337 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.173317    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:38.226192 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.226168    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ce346503\""
Nov 01 06:26:38.229653 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.229626    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"]
Nov 01 06:26:38.269910 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.269885    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9t8q\" (UniqueName: \"kubernetes.io/projected/178405bc-5e30-4a07-acce-31e5be7a94a3-kube-api-access-z9t8q\") pod \"payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl\" (UID: \"178405bc-5e30-4a07-acce-31e5be7a94a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:38.270043 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.269978    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178405bc-5e30-4a07-acce-31e5be7a94a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl\" (UID: \"178405bc-5e30-4a07-acce-31e5be7a94a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:38.275586 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.275569    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ce346503-dockercfg-ttc6v\""
Nov 01 06:26:38.308577 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.308542    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"]
Nov 01 06:26:38.312364 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.312346    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:38.397854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.397150    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqqbs\" (UniqueName: \"kubernetes.io/projected/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-kube-api-access-cqqbs\") pod \"payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2\" (UID: \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\") " pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:38.397854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.397439    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9t8q\" (UniqueName: \"kubernetes.io/projected/178405bc-5e30-4a07-acce-31e5be7a94a3-kube-api-access-z9t8q\") pod \"payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl\" (UID: \"178405bc-5e30-4a07-acce-31e5be7a94a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:38.398098 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.397780    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178405bc-5e30-4a07-acce-31e5be7a94a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl\" (UID: \"178405bc-5e30-4a07-acce-31e5be7a94a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:38.398098 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.398049    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2\" (UID: \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\") " pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:38.404492 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.404462    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178405bc-5e30-4a07-acce-31e5be7a94a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl\" (UID: \"178405bc-5e30-4a07-acce-31e5be7a94a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:38.468116 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.468085    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"]
Nov 01 06:26:38.490698 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.490671    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9t8q\" (UniqueName: \"kubernetes.io/projected/178405bc-5e30-4a07-acce-31e5be7a94a3-kube-api-access-z9t8q\") pod \"payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl\" (UID: \"178405bc-5e30-4a07-acce-31e5be7a94a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:38.505106 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.505076    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2\" (UID: \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\") " pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:38.505255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.505199    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqqbs\" (UniqueName: \"kubernetes.io/projected/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-kube-api-access-cqqbs\") pod \"payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2\" (UID: \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\") " pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:38.505970 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.505950    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2\" (UID: \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\") " pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:38.566052 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.566026    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqqbs\" (UniqueName: \"kubernetes.io/projected/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-kube-api-access-cqqbs\") pod \"payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2\" (UID: \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\") " pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:38.626686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.626652    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:38.793311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.793237    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:38.990983 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:38.990951    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"]
Nov 01 06:26:39.003871 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:39.003846    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"]
Nov 01 06:26:39.005695 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:26:39.005664    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc33dc5a_d931_4f89_841e_3e6348e1a3a3.slice/crio-01dfdbe5e754e24c30aa8eba12550b9e92a817631cb748261aa1988439d15e7a WatchSource:0}: Error finding container 01dfdbe5e754e24c30aa8eba12550b9e92a817631cb748261aa1988439d15e7a: Status 404 returned error can't find the container with id 01dfdbe5e754e24c30aa8eba12550b9e92a817631cb748261aa1988439d15e7a
Nov 01 06:26:39.011285 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:39.011270    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-7fe684b5a48a: secret "agc-cred-7fe684b5a48a" not found
Nov 01 06:26:39.011333 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:39.011289    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-af808cf4bef1: secret "agc-cred-af808cf4bef1" not found
Nov 01 06:26:39.011333 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:39.011298    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-b2178f695dad: secret "agc-cred-b2178f695dad" not found
Nov 01 06:26:39.011333 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:39.011306    5510 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-7d25db/agc-cred-bdcae79313a2: secret "agc-cred-bdcae79313a2" not found
Nov 01 06:26:39.011333 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:39.011315    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6: [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found, secret "agc-cred-bdcae79313a2" not found]
Nov 01 06:26:39.011462 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:39.011360    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials podName:4483cba7-79fc-4997-b9f6-01b6de27d7fd nodeName:}" failed. No retries permitted until 2026-11-01 06:26:55.011346505 +0000 UTC m=+2451.889319894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd") : [secret "agc-cred-7fe684b5a48a" not found, secret "agc-cred-af808cf4bef1" not found, secret "agc-cred-b2178f695dad" not found, secret "agc-cred-bdcae79313a2" not found]
Nov 01 06:26:39.174637 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:39.174612    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"]
Nov 01 06:26:39.897884 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:39.897275    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl" event={"ID":"178405bc-5e30-4a07-acce-31e5be7a94a3","Type":"ContainerStarted","Data":"9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3"}
Nov 01 06:26:39.897884 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:39.897380    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl" event={"ID":"178405bc-5e30-4a07-acce-31e5be7a94a3","Type":"ContainerStarted","Data":"b6f7697b1a1c2b76c41ce8e89e35bfe08798b985cc9cd7827e885858bd4f4e7e"}
Nov 01 06:26:39.898430 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:39.898195    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:39.908260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:39.901657    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2" event={"ID":"fc33dc5a-d931-4f89-841e-3e6348e1a3a3","Type":"ContainerStarted","Data":"c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475"}
Nov 01 06:26:39.908260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:39.901709    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2" event={"ID":"fc33dc5a-d931-4f89-841e-3e6348e1a3a3","Type":"ContainerStarted","Data":"01dfdbe5e754e24c30aa8eba12550b9e92a817631cb748261aa1988439d15e7a"}
Nov 01 06:26:39.908260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:39.902448    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:26:40.015356 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:40.015300    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl" podStartSLOduration=2.015286148 podStartE2EDuration="2.015286148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:40.013261928 +0000 UTC m=+2436.891235339" watchObservedRunningTime="2026-11-01 06:26:40.015286148 +0000 UTC m=+2436.893259558"
Nov 01 06:26:41.703043 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:41.702990    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2" podStartSLOduration=3.702973802 podStartE2EDuration="3.702973802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:40.144340172 +0000 UTC m=+2437.022313611" watchObservedRunningTime="2026-11-01 06:26:41.702973802 +0000 UTC m=+2438.580947219"
Nov 01 06:26:41.704191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:41.704162    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"]
Nov 01 06:26:41.705617 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:41.705591    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" podUID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:41.754496 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:41.754470    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"]
Nov 01 06:26:43.992165 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:43.992071    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9" podUID="ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" containerName="payload-processing" containerID="cri-o://1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52" gracePeriod=25
Nov 01 06:26:44.274402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.274377    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:26:44.403518 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.403491    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-plugins-config-volume\") pod \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") "
Nov 01 06:26:44.403682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.403528    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-routing-overlay-volume\") pod \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") "
Nov 01 06:26:44.403682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.403650    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-kube-api-access-2fxv7\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-kube-api-access-2fxv7\") pod \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") "
Nov 01 06:26:44.403794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.403731    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-provider-credentials\") pod \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\" (UID: \"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90\") "
Nov 01 06:26:44.403982 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.403954    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-plugins-config-volume" pod "ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" (UID: "ad63f7bd-cc6a-44a2-9065-7c9f905c6c90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:44.404065 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.403990    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-routing-overlay-volume" pod "ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" (UID: "ad63f7bd-cc6a-44a2-9065-7c9f905c6c90"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:44.404283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.404262    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:44.404283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.404278    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:44.405990 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.405968    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-provider-credentials" pod "ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" (UID: "ad63f7bd-cc6a-44a2-9065-7c9f905c6c90"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:44.406096 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.406074    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-kube-api-access-2fxv7" pod "ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" (UID: "ad63f7bd-cc6a-44a2-9065-7c9f905c6c90"). InnerVolumeSpecName "kube-api-access-2fxv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:44.505588 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.505511    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:44.505588 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.505537    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fxv7\" (UniqueName: \"kubernetes.io/projected/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90-kube-api-access-2fxv7\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:44.976468 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.976434    5510 generic.go:356] "Generic (PLEG): container finished" podID="ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" containerID="1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52" exitCode=0
Nov 01 06:26:44.976641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.976509    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"
Nov 01 06:26:44.976641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.976512    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9" event={"ID":"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90","Type":"ContainerDied","Data":"1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52"}
Nov 01 06:26:44.976641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.976566    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9" event={"ID":"ad63f7bd-cc6a-44a2-9065-7c9f905c6c90","Type":"ContainerDied","Data":"eebc34e5b5a26c58841bbda0c37a3fd18c5b769b6c04404c922dfff7edcc5de6"}
Nov 01 06:26:44.976641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.976589    5510 scope.go:122] "RemoveContainer" containerID="1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52"
Nov 01 06:26:44.987188 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.987171    5510 scope.go:122] "RemoveContainer" containerID="1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52"
Nov 01 06:26:44.987476 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:44.987451    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52\": container with ID starting with 1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52 not found: ID does not exist" containerID="1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52"
Nov 01 06:26:44.987582 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:44.987480    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52"} err="failed to get container status \"1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52\": rpc error: code = NotFound desc = could not find container \"1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52\": container with ID starting with 1322fccc25f15c8c311d2ecc5494fe0e549bcc14523fdbcd66d1be4c8720ff52 not found: ID does not exist"
Nov 01 06:26:45.150330 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:45.150295    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"]
Nov 01 06:26:45.199702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:45.199678    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-8f8b69tg9"]
Nov 01 06:26:45.245180 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:45.245099    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" path="/var/lib/kubelet/pods/ad63f7bd-cc6a-44a2-9065-7c9f905c6c90/volumes"
Nov 01 06:26:46.705540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:46.705502    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" podUID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" containerName="payload-processing" containerID="cri-o://08fb7179f23fc9c1bf44db048c314360ba469ba482cb53aa6a85a8ff24fa9f5a" gracePeriod=25
Nov 01 06:26:46.754837 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:46.754782    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" podUID="4483cba7-79fc-4997-b9f6-01b6de27d7fd" containerName="payload-processing" containerID="cri-o://f0d4e6b1ebe52c1a27f854433c29a9982c385c1267dcfbabe1c22a38e56c59b6" gracePeriod=25
Nov 01 06:26:46.993305 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:46.993267    5510 generic.go:356] "Generic (PLEG): container finished" podID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" containerID="08fb7179f23fc9c1bf44db048c314360ba469ba482cb53aa6a85a8ff24fa9f5a" exitCode=0
Nov 01 06:26:46.993457 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:46.993439    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" event={"ID":"fa2b5837-29fb-42dc-a907-6ceb361ad78e","Type":"ContainerDied","Data":"08fb7179f23fc9c1bf44db048c314360ba469ba482cb53aa6a85a8ff24fa9f5a"}
Nov 01 06:26:46.995259 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:46.995231    5510 generic.go:356] "Generic (PLEG): container finished" podID="4483cba7-79fc-4997-b9f6-01b6de27d7fd" containerID="f0d4e6b1ebe52c1a27f854433c29a9982c385c1267dcfbabe1c22a38e56c59b6" exitCode=0
Nov 01 06:26:46.995390 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:46.995378    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" event={"ID":"4483cba7-79fc-4997-b9f6-01b6de27d7fd","Type":"ContainerDied","Data":"f0d4e6b1ebe52c1a27f854433c29a9982c385c1267dcfbabe1c22a38e56c59b6"}
Nov 01 06:26:47.026284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.026259    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:47.049235 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.049214    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:26:47.161142 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161108    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-kube-api-access-gmbsw\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-kube-api-access-gmbsw\") pod \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") "
Nov 01 06:26:47.161344 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161179    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-plugins-config-volume\") pod \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") "
Nov 01 06:26:47.161344 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161224    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-provider-credentials\") pod \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") "
Nov 01 06:26:47.161344 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161274    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-plugins-config-volume\") pod \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") "
Nov 01 06:26:47.161344 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161311    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials\") pod \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") "
Nov 01 06:26:47.161613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161348    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-kube-api-access-q2nxr\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-kube-api-access-q2nxr\") pod \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") "
Nov 01 06:26:47.161613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161379    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-routing-overlay-volume\") pod \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\" (UID: \"fa2b5837-29fb-42dc-a907-6ceb361ad78e\") "
Nov 01 06:26:47.161613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161425    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-routing-overlay-volume\") pod \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\" (UID: \"4483cba7-79fc-4997-b9f6-01b6de27d7fd\") "
Nov 01 06:26:47.161747 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161684    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-plugins-config-volume" pod "4483cba7-79fc-4997-b9f6-01b6de27d7fd" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:47.161857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161816    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.161992 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.161968    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-routing-overlay-volume" pod "4483cba7-79fc-4997-b9f6-01b6de27d7fd" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:47.164098 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.164075    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-provider-credentials" pod "fa2b5837-29fb-42dc-a907-6ceb361ad78e" (UID: "fa2b5837-29fb-42dc-a907-6ceb361ad78e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:47.164356 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.164340    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-plugins-config-volume" pod "fa2b5837-29fb-42dc-a907-6ceb361ad78e" (UID: "fa2b5837-29fb-42dc-a907-6ceb361ad78e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:47.164528 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.164504    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-kube-api-access-gmbsw" pod "fa2b5837-29fb-42dc-a907-6ceb361ad78e" (UID: "fa2b5837-29fb-42dc-a907-6ceb361ad78e"). InnerVolumeSpecName "kube-api-access-gmbsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:47.165218 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.165194    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-routing-overlay-volume" pod "fa2b5837-29fb-42dc-a907-6ceb361ad78e" (UID: "fa2b5837-29fb-42dc-a907-6ceb361ad78e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:47.166659 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.166634    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials" pod "4483cba7-79fc-4997-b9f6-01b6de27d7fd" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:47.166963 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.166947    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-kube-api-access-q2nxr" pod "4483cba7-79fc-4997-b9f6-01b6de27d7fd" (UID: "4483cba7-79fc-4997-b9f6-01b6de27d7fd"). InnerVolumeSpecName "kube-api-access-q2nxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:47.263145 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.263121    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbsw\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-kube-api-access-gmbsw\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.263255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.263145    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2b5837-29fb-42dc-a907-6ceb361ad78e-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.263255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.263164    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.263255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.263172    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.263255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.263181    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nxr\" (UniqueName: \"kubernetes.io/projected/4483cba7-79fc-4997-b9f6-01b6de27d7fd-kube-api-access-q2nxr\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.263255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.263191    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2b5837-29fb-42dc-a907-6ceb361ad78e-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.263255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:47.263199    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4483cba7-79fc-4997-b9f6-01b6de27d7fd-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.544563 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:26:47.544481    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:26:48.002573 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.002545    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"
Nov 01 06:26:48.003061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.002584    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6" event={"ID":"4483cba7-79fc-4997-b9f6-01b6de27d7fd","Type":"ContainerDied","Data":"7052a56f0b3608d921fba4d05132e83d13ae575597ca73b566a82ad8f3fe3a9d"}
Nov 01 06:26:48.003061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.002640    5510 scope.go:122] "RemoveContainer" containerID="f0d4e6b1ebe52c1a27f854433c29a9982c385c1267dcfbabe1c22a38e56c59b6"
Nov 01 06:26:48.004438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.004417    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"
Nov 01 06:26:48.004532 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.004416    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9" event={"ID":"fa2b5837-29fb-42dc-a907-6ceb361ad78e","Type":"ContainerDied","Data":"3e317829994be431a2e3525b531936e3be6dcb9674c0596e01e497da882cb646"}
Nov 01 06:26:48.012935 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.012918    5510 scope.go:122] "RemoveContainer" containerID="08fb7179f23fc9c1bf44db048c314360ba469ba482cb53aa6a85a8ff24fa9f5a"
Nov 01 06:26:48.167756 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.167724    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"]
Nov 01 06:26:48.217062 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.217033    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-75f6mmtc9"]
Nov 01 06:26:48.280703 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.280639    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"]
Nov 01 06:26:48.284747 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:48.284726    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7d77hvhq6"]
Nov 01 06:26:49.236187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:49.236154    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4483cba7-79fc-4997-b9f6-01b6de27d7fd" path="/var/lib/kubelet/pods/4483cba7-79fc-4997-b9f6-01b6de27d7fd/volumes"
Nov 01 06:26:49.236556 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:49.236545    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" path="/var/lib/kubelet/pods/fa2b5837-29fb-42dc-a907-6ceb361ad78e/volumes"
Nov 01 06:26:50.911368 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:50.911334    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:26:51.958744 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:26:51.958714    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:27:00.142754 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:00.142720    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:27:17.544023 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:27:17.543980    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:27:37.118377 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:37.118341    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:27:40.575939 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.575900    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"]
Nov 01 06:27:40.576800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.576780    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" containerName="payload-processing"
Nov 01 06:27:40.576887 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.576805    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" containerName="payload-processing"
Nov 01 06:27:40.576887 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.576879    5510 cpu_manager.go:414] "containerMap: removing container" podUID="4483cba7-79fc-4997-b9f6-01b6de27d7fd" containerName="payload-processing"
Nov 01 06:27:40.576887 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.576886    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4483cba7-79fc-4997-b9f6-01b6de27d7fd" containerName="payload-processing"
Nov 01 06:27:40.576999 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.576903    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" containerName="payload-processing"
Nov 01 06:27:40.576999 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.576909    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" containerName="payload-processing"
Nov 01 06:27:40.577066 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.577033    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2b5837-29fb-42dc-a907-6ceb361ad78e" containerName="payload-processing"
Nov 01 06:27:40.577066 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.577046    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad63f7bd-cc6a-44a2-9065-7c9f905c6c90" containerName="payload-processing"
Nov 01 06:27:40.577066 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.577056    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4483cba7-79fc-4997-b9f6-01b6de27d7fd" containerName="payload-processing"
Nov 01 06:27:40.580568 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.580551    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:40.623695 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.623667    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgx9\" (UniqueName: \"kubernetes.io/projected/7fbed767-72b7-4db9-aad3-e778d95f5f4d-kube-api-access-ftgx9\") pod \"payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z\" (UID: \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:40.623894 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.623870    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbed767-72b7-4db9-aad3-e778d95f5f4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z\" (UID: \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:40.634955 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.634931    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-745a18\""
Nov 01 06:27:40.721057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.721021    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-745a18-dockercfg-9w7wr\""
Nov 01 06:27:40.723048 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.723022    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"]
Nov 01 06:27:40.725269 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.725227    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbed767-72b7-4db9-aad3-e778d95f5f4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z\" (UID: \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:40.725553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.725533    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgx9\" (UniqueName: \"kubernetes.io/projected/7fbed767-72b7-4db9-aad3-e778d95f5f4d-kube-api-access-ftgx9\") pod \"payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z\" (UID: \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:40.726670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.726647    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbed767-72b7-4db9-aad3-e778d95f5f4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z\" (UID: \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:40.777326 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.777294    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"]
Nov 01 06:27:40.782496 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.782470    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:40.786552 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.786531    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgx9\" (UniqueName: \"kubernetes.io/projected/7fbed767-72b7-4db9-aad3-e778d95f5f4d-kube-api-access-ftgx9\") pod \"payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z\" (UID: \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:40.800176 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.800150    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"]
Nov 01 06:27:40.827516 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.827451    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzfmf\" (UniqueName: \"kubernetes.io/projected/533f1b8d-f031-4281-a031-6aa330912c57-kube-api-access-lzfmf\") pod \"payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj\" (UID: \"533f1b8d-f031-4281-a031-6aa330912c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:40.827636 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.827615    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533f1b8d-f031-4281-a031-6aa330912c57-plugins-config-volume\") pod \"payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj\" (UID: \"533f1b8d-f031-4281-a031-6aa330912c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:40.893968 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.893938    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:40.929246 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.929220    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzfmf\" (UniqueName: \"kubernetes.io/projected/533f1b8d-f031-4281-a031-6aa330912c57-kube-api-access-lzfmf\") pod \"payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj\" (UID: \"533f1b8d-f031-4281-a031-6aa330912c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:40.929374 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.929312    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533f1b8d-f031-4281-a031-6aa330912c57-plugins-config-volume\") pod \"payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj\" (UID: \"533f1b8d-f031-4281-a031-6aa330912c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:40.930033 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.930013    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533f1b8d-f031-4281-a031-6aa330912c57-plugins-config-volume\") pod \"payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj\" (UID: \"533f1b8d-f031-4281-a031-6aa330912c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:40.991349 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:40.991311    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzfmf\" (UniqueName: \"kubernetes.io/projected/533f1b8d-f031-4281-a031-6aa330912c57-kube-api-access-lzfmf\") pod \"payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj\" (UID: \"533f1b8d-f031-4281-a031-6aa330912c57\") " pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:41.108518 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:41.108447    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:41.273821 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:41.273796    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"]
Nov 01 06:27:41.275962 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:27:41.275936    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbed767_72b7_4db9_aad3_e778d95f5f4d.slice/crio-19bba0bf9c3b08eb592c3a40ee5d84f46025a7c8fd611c5fd76ce3232edaa2bd WatchSource:0}: Error finding container 19bba0bf9c3b08eb592c3a40ee5d84f46025a7c8fd611c5fd76ce3232edaa2bd: Status 404 returned error can't find the container with id 19bba0bf9c3b08eb592c3a40ee5d84f46025a7c8fd611c5fd76ce3232edaa2bd
Nov 01 06:27:41.413655 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:41.413583    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z" event={"ID":"7fbed767-72b7-4db9-aad3-e778d95f5f4d","Type":"ContainerStarted","Data":"32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc"}
Nov 01 06:27:41.413655 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:41.413635    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z" event={"ID":"7fbed767-72b7-4db9-aad3-e778d95f5f4d","Type":"ContainerStarted","Data":"19bba0bf9c3b08eb592c3a40ee5d84f46025a7c8fd611c5fd76ce3232edaa2bd"}
Nov 01 06:27:41.413868 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:41.413716    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:41.481412 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:41.481393    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"]
Nov 01 06:27:41.483622 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:27:41.483599    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533f1b8d_f031_4281_a031_6aa330912c57.slice/crio-c24be151ededca8924d9d4dde652d080cfc0a3021d394ef7c4e081e802600415 WatchSource:0}: Error finding container c24be151ededca8924d9d4dde652d080cfc0a3021d394ef7c4e081e802600415: Status 404 returned error can't find the container with id c24be151ededca8924d9d4dde652d080cfc0a3021d394ef7c4e081e802600415
Nov 01 06:27:41.528751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:41.528708    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z" podStartSLOduration=1.5286933029999998 podStartE2EDuration="1.528693303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:41.528068095 +0000 UTC m=+2498.406041505" watchObservedRunningTime="2026-11-01 06:27:41.528693303 +0000 UTC m=+2498.406666712"
Nov 01 06:27:42.422206 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:42.422173    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj" event={"ID":"533f1b8d-f031-4281-a031-6aa330912c57","Type":"ContainerStarted","Data":"576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236"}
Nov 01 06:27:42.422632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:42.422225    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj" event={"ID":"533f1b8d-f031-4281-a031-6aa330912c57","Type":"ContainerStarted","Data":"c24be151ededca8924d9d4dde652d080cfc0a3021d394ef7c4e081e802600415"}
Nov 01 06:27:42.422632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:42.422486    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:27:42.540944 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:42.540878    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj" podStartSLOduration=2.540860679 podStartE2EDuration="2.540860679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:42.539997188 +0000 UTC m=+2499.417970598" watchObservedRunningTime="2026-11-01 06:27:42.540860679 +0000 UTC m=+2499.418834092"
Nov 01 06:27:47.544596 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:27:47.544559    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:27:52.454677 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:52.454646    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:27:53.433809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:27:53.433779    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:28:13.832117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.832045    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-777477558b-jh8n4"]
Nov 01 06:28:13.843211 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.843177    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:13.894247 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.894216    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-777477558b-jh8n4"]
Nov 01 06:28:13.906560 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.906533    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:13.906689 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.906607    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxdb\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-kube-api-access-gvxdb\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:13.906756 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.906736    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-routing-overlay-volume\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:13.906797 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.906767    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-plugins-config-volume\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:13.939843 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.939788    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6450c221506c\""
Nov 01 06:28:13.939843 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.939809    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 06:28:13.939843 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.939807    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 06:28:13.940036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.939890    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vlbhl\""
Nov 01 06:28:13.944433 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.944396    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 06:28:13.944564 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:13.944467    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 06:28:14.007878 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.007853    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxdb\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-kube-api-access-gvxdb\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.008001 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.007963    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-routing-overlay-volume\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.008001 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.007985    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-plugins-config-volume\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.008087 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.008059    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.008667 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.008642    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-routing-overlay-volume\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.008779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.008707    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-plugins-config-volume\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.010532 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.010509    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.067766 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.067740    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxdb\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-kube-api-access-gvxdb\") pod \"payload-processing-external-model-777477558b-jh8n4\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") " pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.156566 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.156493    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.557015 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.555651    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-777477558b-jh8n4"]
Nov 01 06:28:14.673913 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.673879    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4" event={"ID":"c291622a-19c7-4c34-9e04-0675ef64753f","Type":"ContainerStarted","Data":"64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae"}
Nov 01 06:28:14.674095 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.673932    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4" event={"ID":"c291622a-19c7-4c34-9e04-0675ef64753f","Type":"ContainerStarted","Data":"1705321d410229f8894c523fb7b76bf64ebdc73d808a2340f09e2337c38ea719"}
Nov 01 06:28:14.674095 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.674070    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:14.790322 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:14.790222    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4" podStartSLOduration=1.790207606 podStartE2EDuration="1.790207606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:14.788472269 +0000 UTC m=+2531.666445680" watchObservedRunningTime="2026-11-01 06:28:14.790207606 +0000 UTC m=+2531.668181018"
Nov 01 06:28:15.732924 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:15.732886    5510 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6450c221506c: secret "agc-cred-6450c221506c" not found
Nov 01 06:28:15.732924 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:15.732917    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777477558b-jh8n4: secret "agc-cred-6450c221506c" not found
Nov 01 06:28:15.733387 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:15.732998    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials podName:c291622a-19c7-4c34-9e04-0675ef64753f nodeName:}" failed. No retries permitted until 2026-11-01 06:28:16.23297693 +0000 UTC m=+2533.110950333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials") pod "payload-processing-external-model-777477558b-jh8n4" (UID: "c291622a-19c7-4c34-9e04-0675ef64753f") : secret "agc-cred-6450c221506c" not found
Nov 01 06:28:16.238302 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:16.238273    5510 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6450c221506c: secret "agc-cred-6450c221506c" not found
Nov 01 06:28:16.238302 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:16.238296    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777477558b-jh8n4: secret "agc-cred-6450c221506c" not found
Nov 01 06:28:16.238513 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:16.238345    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials podName:c291622a-19c7-4c34-9e04-0675ef64753f nodeName:}" failed. No retries permitted until 2026-11-01 06:28:17.238331656 +0000 UTC m=+2534.116305043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials") pod "payload-processing-external-model-777477558b-jh8n4" (UID: "c291622a-19c7-4c34-9e04-0675ef64753f") : secret "agc-cred-6450c221506c" not found
Nov 01 06:28:16.615410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:16.615332    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-777477558b-jh8n4"]
Nov 01 06:28:17.254900 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:17.254863    5510 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6450c221506c: secret "agc-cred-6450c221506c" not found
Nov 01 06:28:17.255485 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:17.255467    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777477558b-jh8n4: secret "agc-cred-6450c221506c" not found
Nov 01 06:28:17.255649 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:17.255637    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials podName:c291622a-19c7-4c34-9e04-0675ef64753f nodeName:}" failed. No retries permitted until 2026-11-01 06:28:19.25561239 +0000 UTC m=+2536.133585781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials") pod "payload-processing-external-model-777477558b-jh8n4" (UID: "c291622a-19c7-4c34-9e04-0675ef64753f") : secret "agc-cred-6450c221506c" not found
Nov 01 06:28:17.558263 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:17.558170    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:28:19.282108 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:19.282071    5510 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6450c221506c: secret "agc-cred-6450c221506c" not found
Nov 01 06:28:19.282108 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:19.282095    5510 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777477558b-jh8n4: secret "agc-cred-6450c221506c" not found
Nov 01 06:28:19.282587 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:19.282157    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials podName:c291622a-19c7-4c34-9e04-0675ef64753f nodeName:}" failed. No retries permitted until 2026-11-01 06:28:23.282142324 +0000 UTC m=+2540.160115729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials") pod "payload-processing-external-model-777477558b-jh8n4" (UID: "c291622a-19c7-4c34-9e04-0675ef64753f") : secret "agc-cred-6450c221506c" not found
Nov 01 06:28:21.686493 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:21.686454    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4" podUID="c291622a-19c7-4c34-9e04-0675ef64753f" containerName="payload-processing" containerID="cri-o://64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae" gracePeriod=25
Nov 01 06:28:21.955943 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:21.955919    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:22.113950 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.113921    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-plugins-config-volume\") pod \"c291622a-19c7-4c34-9e04-0675ef64753f\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") "
Nov 01 06:28:22.114107 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.114025    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials\") pod \"c291622a-19c7-4c34-9e04-0675ef64753f\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") "
Nov 01 06:28:22.114107 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.114056    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-kube-api-access-gvxdb\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-kube-api-access-gvxdb\") pod \"c291622a-19c7-4c34-9e04-0675ef64753f\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") "
Nov 01 06:28:22.114216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.114123    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-routing-overlay-volume\") pod \"c291622a-19c7-4c34-9e04-0675ef64753f\" (UID: \"c291622a-19c7-4c34-9e04-0675ef64753f\") "
Nov 01 06:28:22.114336 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.114303    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-plugins-config-volume" pod "c291622a-19c7-4c34-9e04-0675ef64753f" (UID: "c291622a-19c7-4c34-9e04-0675ef64753f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:22.114539 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.114516    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-routing-overlay-volume" pod "c291622a-19c7-4c34-9e04-0675ef64753f" (UID: "c291622a-19c7-4c34-9e04-0675ef64753f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:22.114685 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.114666    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:22.114733 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.114688    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c291622a-19c7-4c34-9e04-0675ef64753f-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:22.116414 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.116398    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials" pod "c291622a-19c7-4c34-9e04-0675ef64753f" (UID: "c291622a-19c7-4c34-9e04-0675ef64753f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:22.116464 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.116411    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-kube-api-access-gvxdb" pod "c291622a-19c7-4c34-9e04-0675ef64753f" (UID: "c291622a-19c7-4c34-9e04-0675ef64753f"). InnerVolumeSpecName "kube-api-access-gvxdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:22.216235 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.216207    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxdb\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-kube-api-access-gvxdb\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:22.216235 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.216231    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c291622a-19c7-4c34-9e04-0675ef64753f-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:22.753936 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.753899    5510 generic.go:356] "Generic (PLEG): container finished" podID="c291622a-19c7-4c34-9e04-0675ef64753f" containerID="64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae" exitCode=0
Nov 01 06:28:22.754663 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.754634    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4" event={"ID":"c291622a-19c7-4c34-9e04-0675ef64753f","Type":"ContainerDied","Data":"64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae"}
Nov 01 06:28:22.754867 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.754848    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4" event={"ID":"c291622a-19c7-4c34-9e04-0675ef64753f","Type":"ContainerDied","Data":"1705321d410229f8894c523fb7b76bf64ebdc73d808a2340f09e2337c38ea719"}
Nov 01 06:28:22.755094 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.754965    5510 scope.go:122] "RemoveContainer" containerID="64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae"
Nov 01 06:28:22.755094 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.754948    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-777477558b-jh8n4"
Nov 01 06:28:22.785872 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.785816    5510 scope.go:122] "RemoveContainer" containerID="64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae"
Nov 01 06:28:22.786243 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:22.786207    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae\": container with ID starting with 64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae not found: ID does not exist" containerID="64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae"
Nov 01 06:28:22.786303 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.786244    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae"} err="failed to get container status \"64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae\": rpc error: code = NotFound desc = could not find container \"64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae\": container with ID starting with 64654a9c1fbacad17be378586cf7233adb90d5686cbb8d5690344a5762193eae not found: ID does not exist"
Nov 01 06:28:22.918316 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.918280    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-777477558b-jh8n4"]
Nov 01 06:28:22.949366 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:22.949338    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-777477558b-jh8n4"]
Nov 01 06:28:23.236111 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:23.236081    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c291622a-19c7-4c34-9e04-0675ef64753f" path="/var/lib/kubelet/pods/c291622a-19c7-4c34-9e04-0675ef64753f/volumes"
Nov 01 06:28:31.881009 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:31.880977    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"]
Nov 01 06:28:31.881583 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:31.881559    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c291622a-19c7-4c34-9e04-0675ef64753f" containerName="payload-processing"
Nov 01 06:28:31.881640 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:31.881585    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c291622a-19c7-4c34-9e04-0675ef64753f" containerName="payload-processing"
Nov 01 06:28:31.881708 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:31.881698    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c291622a-19c7-4c34-9e04-0675ef64753f" containerName="payload-processing"
Nov 01 06:28:31.885147 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:31.885130    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:31.942901 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:31.942741    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-745a18-serving-cert\""
Nov 01 06:28:31.996670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:31.996629    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"]
Nov 01 06:28:32.030301 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.030271    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0891aff-10dc-4366-87dc-5e1e3afe935f-openshift-service-ca\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.030466 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.030424    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-metrics-certs\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.030528 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.030479    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzfm\" (UniqueName: \"kubernetes.io/projected/d0891aff-10dc-4366-87dc-5e1e3afe935f-kube-api-access-ngzfm\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.030662 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.030642    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-maas-api-tls\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.131937 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.131855    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-metrics-certs\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.131937 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.131907    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzfm\" (UniqueName: \"kubernetes.io/projected/d0891aff-10dc-4366-87dc-5e1e3afe935f-kube-api-access-ngzfm\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.132114 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.131977    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-maas-api-tls\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.132114 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.132014    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0891aff-10dc-4366-87dc-5e1e3afe935f-openshift-service-ca\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.132714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.132650    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0891aff-10dc-4366-87dc-5e1e3afe935f-openshift-service-ca\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.134745 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.134719    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-metrics-certs\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.135348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.135330    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-maas-api-tls\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.195288 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.195256    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzfm\" (UniqueName: \"kubernetes.io/projected/d0891aff-10dc-4366-87dc-5e1e3afe935f-kube-api-access-ngzfm\") pod \"maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.198147 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.198121    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:32.222609 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.222586    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:28:32.638800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.638766    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"]
Nov 01 06:28:32.640506 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:28:32.640482    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0891aff_10dc_4366_87dc_5e1e3afe935f.slice/crio-3f9ca620acff69def3bf4054a5152668339a69a09222d961d6916a6d5c90797c WatchSource:0}: Error finding container 3f9ca620acff69def3bf4054a5152668339a69a09222d961d6916a6d5c90797c: Status 404 returned error can't find the container with id 3f9ca620acff69def3bf4054a5152668339a69a09222d961d6916a6d5c90797c
Nov 01 06:28:32.835353 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:32.835320    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk" event={"ID":"d0891aff-10dc-4366-87dc-5e1e3afe935f","Type":"ContainerStarted","Data":"3f9ca620acff69def3bf4054a5152668339a69a09222d961d6916a6d5c90797c"}
Nov 01 06:28:33.863088 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:33.862986    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk" event={"ID":"d0891aff-10dc-4366-87dc-5e1e3afe935f","Type":"ContainerStarted","Data":"77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15"}
Nov 01 06:28:33.863663 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:33.863641    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:33.988505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:33.988431    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk" podStartSLOduration=2.061802715 podStartE2EDuration="2.988411742s" totalImagesPullingTime="926.609027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:33.985352388 +0000 UTC m=+2550.863325796" watchObservedRunningTime="2026-11-01 06:28:33.988411742 +0000 UTC m=+2550.866385152"
Nov 01 06:28:34.466961 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:34.466926    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:28:34.467158 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:34.466984    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:28:39.873387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:39.873347    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:28:47.544795 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:47.544757    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:28:50.193563 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:50.193532    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"]
Nov 01 06:28:50.297585 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:50.297541    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"]
Nov 01 06:28:55.194180 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.194144    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2" podUID="fc33dc5a-d931-4f89-841e-3e6348e1a3a3" containerName="payload-processing" containerID="cri-o://c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475" gracePeriod=25
Nov 01 06:28:55.298693 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.298608    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl" podUID="178405bc-5e30-4a07-acce-31e5be7a94a3" containerName="payload-pre-processing" containerID="cri-o://9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3" gracePeriod=25
Nov 01 06:28:55.464066 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.464038    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:28:55.488220 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.488193    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-kube-api-access-cqqbs\" (UniqueName: \"kubernetes.io/projected/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-kube-api-access-cqqbs\") pod \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\" (UID: \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\") "
Nov 01 06:28:55.488366 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.488236    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-plugins-config-volume\") pod \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\" (UID: \"fc33dc5a-d931-4f89-841e-3e6348e1a3a3\") "
Nov 01 06:28:55.488649 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.488625    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-plugins-config-volume" pod "fc33dc5a-d931-4f89-841e-3e6348e1a3a3" (UID: "fc33dc5a-d931-4f89-841e-3e6348e1a3a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:55.488712 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.488702    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:55.490940 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.490909    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-kube-api-access-cqqbs" pod "fc33dc5a-d931-4f89-841e-3e6348e1a3a3" (UID: "fc33dc5a-d931-4f89-841e-3e6348e1a3a3"). InnerVolumeSpecName "kube-api-access-cqqbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:55.555284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.555228    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:28:55.589243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.589216    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178405bc-5e30-4a07-acce-31e5be7a94a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178405bc-5e30-4a07-acce-31e5be7a94a3-plugins-config-volume\") pod \"178405bc-5e30-4a07-acce-31e5be7a94a3\" (UID: \"178405bc-5e30-4a07-acce-31e5be7a94a3\") "
Nov 01 06:28:55.589400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.589355    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178405bc-5e30-4a07-acce-31e5be7a94a3-kube-api-access-z9t8q\" (UniqueName: \"kubernetes.io/projected/178405bc-5e30-4a07-acce-31e5be7a94a3-kube-api-access-z9t8q\") pod \"178405bc-5e30-4a07-acce-31e5be7a94a3\" (UID: \"178405bc-5e30-4a07-acce-31e5be7a94a3\") "
Nov 01 06:28:55.589654 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.589628    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178405bc-5e30-4a07-acce-31e5be7a94a3-plugins-config-volume" pod "178405bc-5e30-4a07-acce-31e5be7a94a3" (UID: "178405bc-5e30-4a07-acce-31e5be7a94a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:55.589811 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.589792    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqqbs\" (UniqueName: \"kubernetes.io/projected/fc33dc5a-d931-4f89-841e-3e6348e1a3a3-kube-api-access-cqqbs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:55.589879 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.589818    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178405bc-5e30-4a07-acce-31e5be7a94a3-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:55.591498 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.591477    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178405bc-5e30-4a07-acce-31e5be7a94a3-kube-api-access-z9t8q" pod "178405bc-5e30-4a07-acce-31e5be7a94a3" (UID: "178405bc-5e30-4a07-acce-31e5be7a94a3"). InnerVolumeSpecName "kube-api-access-z9t8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:55.691452 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:55.691422    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9t8q\" (UniqueName: \"kubernetes.io/projected/178405bc-5e30-4a07-acce-31e5be7a94a3-kube-api-access-z9t8q\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:56.016953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.016916    5510 generic.go:356] "Generic (PLEG): container finished" podID="fc33dc5a-d931-4f89-841e-3e6348e1a3a3" containerID="c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475" exitCode=0
Nov 01 06:28:56.017127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.016963    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"
Nov 01 06:28:56.017127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.017005    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2" event={"ID":"fc33dc5a-d931-4f89-841e-3e6348e1a3a3","Type":"ContainerDied","Data":"c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475"}
Nov 01 06:28:56.017127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.017054    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2" event={"ID":"fc33dc5a-d931-4f89-841e-3e6348e1a3a3","Type":"ContainerDied","Data":"01dfdbe5e754e24c30aa8eba12550b9e92a817631cb748261aa1988439d15e7a"}
Nov 01 06:28:56.017127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.017100    5510 scope.go:122] "RemoveContainer" containerID="c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475"
Nov 01 06:28:56.019026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.019005    5510 generic.go:356] "Generic (PLEG): container finished" podID="178405bc-5e30-4a07-acce-31e5be7a94a3" containerID="9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3" exitCode=0
Nov 01 06:28:56.019167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.019060    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"
Nov 01 06:28:56.019167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.019068    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl" event={"ID":"178405bc-5e30-4a07-acce-31e5be7a94a3","Type":"ContainerDied","Data":"9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3"}
Nov 01 06:28:56.019167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.019101    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl" event={"ID":"178405bc-5e30-4a07-acce-31e5be7a94a3","Type":"ContainerDied","Data":"b6f7697b1a1c2b76c41ce8e89e35bfe08798b985cc9cd7827e885858bd4f4e7e"}
Nov 01 06:28:56.030915 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.030867    5510 scope.go:122] "RemoveContainer" containerID="c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475"
Nov 01 06:28:56.031288 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:56.031259    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475\": container with ID starting with c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475 not found: ID does not exist" containerID="c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475"
Nov 01 06:28:56.031369 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.031303    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475"} err="failed to get container status \"c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475\": rpc error: code = NotFound desc = could not find container \"c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475\": container with ID starting with c7a5f3f6593efbbc22b3b7f4763c5a69e8f05447a0a389d4ad74b8ba6ce3b475 not found: ID does not exist"
Nov 01 06:28:56.031369 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.031327    5510 scope.go:122] "RemoveContainer" containerID="9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3"
Nov 01 06:28:56.042786 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.042764    5510 scope.go:122] "RemoveContainer" containerID="9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3"
Nov 01 06:28:56.043082 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:28:56.043064    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3\": container with ID starting with 9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3 not found: ID does not exist" containerID="9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3"
Nov 01 06:28:56.043148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.043089    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3"} err="failed to get container status \"9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3\": rpc error: code = NotFound desc = could not find container \"9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3\": container with ID starting with 9ccfeb0ebaae1afa53f0e91a6c9be32177183dbb23b3799c9b018e289c23e8c3 not found: ID does not exist"
Nov 01 06:28:56.183481 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.183449    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"]
Nov 01 06:28:56.235035 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.235005    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce346503-7986577c5c-dxlg2"]
Nov 01 06:28:56.296809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.296725    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"]
Nov 01 06:28:56.301778 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:56.301754    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce346503-6976877875-wtvxl"]
Nov 01 06:28:57.235861 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:57.235810    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178405bc-5e30-4a07-acce-31e5be7a94a3" path="/var/lib/kubelet/pods/178405bc-5e30-4a07-acce-31e5be7a94a3/volumes"
Nov 01 06:28:57.236238 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:28:57.236226    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc33dc5a-d931-4f89-841e-3e6348e1a3a3" path="/var/lib/kubelet/pods/fc33dc5a-d931-4f89-841e-3e6348e1a3a3/volumes"
Nov 01 06:29:06.037818 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:06.037781    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:29:17.545333 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:29:17.545293    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:29:33.909625 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.909590    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"]
Nov 01 06:29:33.910248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.910232    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fc33dc5a-d931-4f89-841e-3e6348e1a3a3" containerName="payload-processing"
Nov 01 06:29:33.910292 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.910249    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc33dc5a-d931-4f89-841e-3e6348e1a3a3" containerName="payload-processing"
Nov 01 06:29:33.910292 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.910286    5510 cpu_manager.go:414] "containerMap: removing container" podUID="178405bc-5e30-4a07-acce-31e5be7a94a3" containerName="payload-pre-processing"
Nov 01 06:29:33.910355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.910292    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178405bc-5e30-4a07-acce-31e5be7a94a3" containerName="payload-pre-processing"
Nov 01 06:29:33.910403 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.910394    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc33dc5a-d931-4f89-841e-3e6348e1a3a3" containerName="payload-processing"
Nov 01 06:29:33.910437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.910407    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="178405bc-5e30-4a07-acce-31e5be7a94a3" containerName="payload-pre-processing"
Nov 01 06:29:33.913857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.913817    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:33.965705 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:33.965636    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0aed62e2-openshift-default-dockercfg-b8mcp\""
Nov 01 06:29:34.039960 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.038238    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"]
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.048994    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-podinfo\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.049070    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-certs\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.049106    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-envoy\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.049193    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22g6q\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-kube-api-access-22g6q\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.049226    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istiod-ca-cert\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.049294    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-token\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.049380    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-socket\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.049407    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-credential-socket\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.049505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.049449    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-data\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150170 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150139    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-podinfo\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150367 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150193    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-certs\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150447 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150367    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-envoy\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150466    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22g6q\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-kube-api-access-22g6q\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150501    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istiod-ca-cert\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150568    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-token\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150608    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-certs\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150670    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-socket\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150710    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-credential-socket\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.150817 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.150762    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-data\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.151042 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.151023    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-socket\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.151185 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.151162    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-credential-socket\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.151381 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.151356    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-data\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.151423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.151370    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istiod-ca-cert\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.152891 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.152868    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-envoy\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.153031 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.153011    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-podinfo\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.212022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.211991    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-token\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.212232 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.212210    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22g6q\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-kube-api-access-22g6q\") pod \"e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") " pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.226485 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.226457    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:34.660138 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.660113    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"]
Nov 01 06:29:34.661610 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:29:34.661588    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3c5211_1885_4fbd_a6f9_da8f8ac2c7e5.slice/crio-b74d9b13e9bdd5b3ee9944bea883f773533130551fdf02102bc11b7e545ae99a WatchSource:0}: Error finding container b74d9b13e9bdd5b3ee9944bea883f773533130551fdf02102bc11b7e545ae99a: Status 404 returned error can't find the container with id b74d9b13e9bdd5b3ee9944bea883f773533130551fdf02102bc11b7e545ae99a
Nov 01 06:29:34.663967 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.663932    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:29:34.664057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.663997    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:29:34.664057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:34.664027    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:29:35.252058 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:35.252026    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" event={"ID":"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5","Type":"ContainerStarted","Data":"69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784"}
Nov 01 06:29:35.252481 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:35.252072    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" event={"ID":"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5","Type":"ContainerStarted","Data":"b74d9b13e9bdd5b3ee9944bea883f773533130551fdf02102bc11b7e545ae99a"}
Nov 01 06:29:35.373063 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:35.373015    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" podStartSLOduration=2.373001781 podStartE2EDuration="2.373001781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:35.371218403 +0000 UTC m=+2612.249191825" watchObservedRunningTime="2026-11-01 06:29:35.373001781 +0000 UTC m=+2612.250975190"
Nov 01 06:29:36.227050 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:36.227016    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:36.233085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:36.233062    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:36.257442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:36.257411    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:36.258126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:36.258107    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:29:42.979850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:42.969121    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"]
Nov 01 06:29:42.980381 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:42.980104    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:43.028634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.028604    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"]
Nov 01 06:29:43.075074 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.075047    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0aed62e2\""
Nov 01 06:29:43.075074 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.075074    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0aed62e2-dockercfg-qjmzt\""
Nov 01 06:29:43.171545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.171511    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-plugins-config-volume\") pod \"payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb\" (UID: \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\") " pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:43.171744 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.171703    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2v9\" (UniqueName: \"kubernetes.io/projected/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-kube-api-access-zm2v9\") pod \"payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb\" (UID: \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\") " pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:43.272822 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.272748    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2v9\" (UniqueName: \"kubernetes.io/projected/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-kube-api-access-zm2v9\") pod \"payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb\" (UID: \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\") " pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:43.272993 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.272867    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-plugins-config-volume\") pod \"payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb\" (UID: \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\") " pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:43.273559 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.273539    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-plugins-config-volume\") pod \"payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb\" (UID: \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\") " pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:43.333259 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.333231    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2v9\" (UniqueName: \"kubernetes.io/projected/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-kube-api-access-zm2v9\") pod \"payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb\" (UID: \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\") " pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:43.599993 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.599895    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:43.978784 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:43.978757    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"]
Nov 01 06:29:43.980491 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:29:43.980464    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591cae75_2f5a_4c11_8ea6_1cfc931e3eec.slice/crio-55c27a5ae4a1fb944e71ea2c3fd8618816088b22d57b97ae5c1460e1ddbc436c WatchSource:0}: Error finding container 55c27a5ae4a1fb944e71ea2c3fd8618816088b22d57b97ae5c1460e1ddbc436c: Status 404 returned error can't find the container with id 55c27a5ae4a1fb944e71ea2c3fd8618816088b22d57b97ae5c1460e1ddbc436c
Nov 01 06:29:44.346990 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:44.346896    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb" event={"ID":"591cae75-2f5a-4c11-8ea6-1cfc931e3eec","Type":"ContainerStarted","Data":"9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d"}
Nov 01 06:29:44.346990 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:44.346951    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb" event={"ID":"591cae75-2f5a-4c11-8ea6-1cfc931e3eec","Type":"ContainerStarted","Data":"55c27a5ae4a1fb944e71ea2c3fd8618816088b22d57b97ae5c1460e1ddbc436c"}
Nov 01 06:29:44.347196 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:44.347098    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:29:44.413589 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:44.413549    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:29:47.546370 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:29:47.546333    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:29:51.822535 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.822484    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb" podStartSLOduration=9.822466163 podStartE2EDuration="9.822466163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:44.478137093 +0000 UTC m=+2621.356110502" watchObservedRunningTime="2026-11-01 06:29:51.822466163 +0000 UTC m=+2628.700439572"
Nov 01 06:29:51.823462 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.823443    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"]
Nov 01 06:29:51.827776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.827750    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:51.900877 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.900843    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28s9w\" (UniqueName: \"kubernetes.io/projected/44b495b2-6210-4d04-845d-e942c97c4a3f-kube-api-access-28s9w\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:51.901017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.900958    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-metrics-certs\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:51.901070 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.901027    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:51.901070 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.901050    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b495b2-6210-4d04-845d-e942c97c4a3f-openshift-service-ca\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:51.932097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.932073    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c0662f25-serving-cert\""
Nov 01 06:29:51.934706 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:51.934686    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"]
Nov 01 06:29:52.002209 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.002182    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28s9w\" (UniqueName: \"kubernetes.io/projected/44b495b2-6210-4d04-845d-e942c97c4a3f-kube-api-access-28s9w\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.002384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.002250    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-metrics-certs\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.002384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.002306    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.002384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.002326    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b495b2-6210-4d04-845d-e942c97c4a3f-openshift-service-ca\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.002563 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:29:52.002417    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-serving-cert: secret "maas-api-e2e-mt-c0662f25-serving-cert" not found
Nov 01 06:29:52.002563 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:29:52.002506    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls podName:44b495b2-6210-4d04-845d-e942c97c4a3f nodeName:}" failed. No retries permitted until 2026-11-01 06:29:52.502481798 +0000 UTC m=+2629.380455204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls") pod "maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf" (UID: "44b495b2-6210-4d04-845d-e942c97c4a3f") : secret "maas-api-e2e-mt-c0662f25-serving-cert" not found
Nov 01 06:29:52.002967 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.002949    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b495b2-6210-4d04-845d-e942c97c4a3f-openshift-service-ca\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.004798 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.004777    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-metrics-certs\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.063909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.063877    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28s9w\" (UniqueName: \"kubernetes.io/projected/44b495b2-6210-4d04-845d-e942c97c4a3f-kube-api-access-28s9w\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.510352 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.510317    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.512955 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.512934    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:52.740974 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:52.740938    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:53.121121 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:53.121094    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"]
Nov 01 06:29:53.122484 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:29:53.122451    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b495b2_6210_4d04_845d_e942c97c4a3f.slice/crio-a4aecf4e5afd095cf6bc73f876651f6a33f982f65ed0c4b72b101819cfeb23d6 WatchSource:0}: Error finding container a4aecf4e5afd095cf6bc73f876651f6a33f982f65ed0c4b72b101819cfeb23d6: Status 404 returned error can't find the container with id a4aecf4e5afd095cf6bc73f876651f6a33f982f65ed0c4b72b101819cfeb23d6
Nov 01 06:29:53.123838 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:53.123804    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:29:53.402206 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:53.402127    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf" event={"ID":"44b495b2-6210-4d04-845d-e942c97c4a3f","Type":"ContainerStarted","Data":"a4aecf4e5afd095cf6bc73f876651f6a33f982f65ed0c4b72b101819cfeb23d6"}
Nov 01 06:29:54.410273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:54.410238    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf" event={"ID":"44b495b2-6210-4d04-845d-e942c97c4a3f","Type":"ContainerStarted","Data":"bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23"}
Nov 01 06:29:54.410650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:54.410302    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:29:54.528910 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:54.528856    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf" podStartSLOduration=2.709567142 podStartE2EDuration="3.528840363s" totalImagesPullingTime="819.273221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:54.52713159 +0000 UTC m=+2631.405104996" watchObservedRunningTime="2026-11-01 06:29:54.528840363 +0000 UTC m=+2631.406813776"
Nov 01 06:29:55.356534 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:29:55.356454    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:30:00.192674 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.192638    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"]
Nov 01 06:30:00.199840 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.199809    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"]
Nov 01 06:30:00.199971 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.199938    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:00.203478 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.203459    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:00.212920 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.212895    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"]
Nov 01 06:30:00.217437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.217415    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:00.238063 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.238038    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"]
Nov 01 06:30:00.241981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.241963    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:00.308592 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.308557    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dblhg\" (UniqueName: \"kubernetes.io/projected/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-kube-api-access-dblhg\") pod \"maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d\" (UID: \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:00.308858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.308822    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxs4w\" (UniqueName: \"kubernetes.io/projected/0683ea85-a108-4759-a595-19434ec813d4-kube-api-access-bxs4w\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db\" (UID: \"0683ea85-a108-4759-a595-19434ec813d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:00.308979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.308966    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fpwq\" (UniqueName: \"kubernetes.io/projected/9e224c5d-28de-49b9-9d13-1ea12e4b3628-kube-api-access-9fpwq\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl\" (UID: \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:00.309080 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.309069    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e224c5d-28de-49b9-9d13-1ea12e4b3628-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl\" (UID: \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:00.309183 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.309170    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8865\" (UniqueName: \"kubernetes.io/projected/453beafd-45c7-4526-895d-2bebf0f9aad4-kube-api-access-d8865\") pod \"maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd\" (UID: \"453beafd-45c7-4526-895d-2bebf0f9aad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:00.309270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.309259    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d\" (UID: \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:00.309384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.309373    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453beafd-45c7-4526-895d-2bebf0f9aad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd\" (UID: \"453beafd-45c7-4526-895d-2bebf0f9aad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:00.309572 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.309559    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683ea85-a108-4759-a595-19434ec813d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db\" (UID: \"0683ea85-a108-4759-a595-19434ec813d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:00.329269 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.329233    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:30:00.340918 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.340890    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"]
Nov 01 06:30:00.347873 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.347842    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"]
Nov 01 06:30:00.373256 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.373227    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"]
Nov 01 06:30:00.375512 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.375487    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"]
Nov 01 06:30:00.410800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.410771    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683ea85-a108-4759-a595-19434ec813d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db\" (UID: \"0683ea85-a108-4759-a595-19434ec813d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:00.411052 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.410974    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblhg\" (UniqueName: \"kubernetes.io/projected/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-kube-api-access-dblhg\") pod \"maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d\" (UID: \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:00.411151 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.411065    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxs4w\" (UniqueName: \"kubernetes.io/projected/0683ea85-a108-4759-a595-19434ec813d4-kube-api-access-bxs4w\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db\" (UID: \"0683ea85-a108-4759-a595-19434ec813d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:00.411151 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.411113    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fpwq\" (UniqueName: \"kubernetes.io/projected/9e224c5d-28de-49b9-9d13-1ea12e4b3628-kube-api-access-9fpwq\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl\" (UID: \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:00.411151 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.411144    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e224c5d-28de-49b9-9d13-1ea12e4b3628-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl\" (UID: \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:00.411317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.411180    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8865\" (UniqueName: \"kubernetes.io/projected/453beafd-45c7-4526-895d-2bebf0f9aad4-kube-api-access-d8865\") pod \"maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd\" (UID: \"453beafd-45c7-4526-895d-2bebf0f9aad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:00.411317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.411205    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d\" (UID: \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:00.411317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.411252    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453beafd-45c7-4526-895d-2bebf0f9aad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd\" (UID: \"453beafd-45c7-4526-895d-2bebf0f9aad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:00.411718 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.411692    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683ea85-a108-4759-a595-19434ec813d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db\" (UID: \"0683ea85-a108-4759-a595-19434ec813d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:00.412004 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.411978    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453beafd-45c7-4526-895d-2bebf0f9aad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd\" (UID: \"453beafd-45c7-4526-895d-2bebf0f9aad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:00.412124 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.412107    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d\" (UID: \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:00.412267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.412242    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e224c5d-28de-49b9-9d13-1ea12e4b3628-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl\" (UID: \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:00.423937 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.423917    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:30:00.473462 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.473433    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fpwq\" (UniqueName: \"kubernetes.io/projected/9e224c5d-28de-49b9-9d13-1ea12e4b3628-kube-api-access-9fpwq\") pod \"maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl\" (UID: \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:00.473635 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.473623    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblhg\" (UniqueName: \"kubernetes.io/projected/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-kube-api-access-dblhg\") pod \"maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d\" (UID: \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:00.473853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.473811    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxs4w\" (UniqueName: \"kubernetes.io/projected/0683ea85-a108-4759-a595-19434ec813d4-kube-api-access-bxs4w\") pod \"maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db\" (UID: \"0683ea85-a108-4759-a595-19434ec813d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:00.473900 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.473811    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8865\" (UniqueName: \"kubernetes.io/projected/453beafd-45c7-4526-895d-2bebf0f9aad4-kube-api-access-d8865\") pod \"maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd\" (UID: \"453beafd-45c7-4526-895d-2bebf0f9aad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:00.516858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.516838    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:00.523930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.523910    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:00.531631 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.531613    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:00.555283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.555256    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:00.909891 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.909858    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"]
Nov 01 06:30:00.954819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.954782    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"]
Nov 01 06:30:00.955448 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:30:00.955413    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0683ea85_a108_4759_a595_19434ec813d4.slice/crio-8d310e4730ab6ec6b6321c83654995514a7e68cbc913f7d508ff559b3de97d97 WatchSource:0}: Error finding container 8d310e4730ab6ec6b6321c83654995514a7e68cbc913f7d508ff559b3de97d97: Status 404 returned error can't find the container with id 8d310e4730ab6ec6b6321c83654995514a7e68cbc913f7d508ff559b3de97d97
Nov 01 06:30:00.959204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.959174    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"]
Nov 01 06:30:00.959587 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:30:00.959567    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453beafd_45c7_4526_895d_2bebf0f9aad4.slice/crio-d35de4d743d724d234350ae63e00d3c64b630bf8d0809914a1fff25220de2423 WatchSource:0}: Error finding container d35de4d743d724d234350ae63e00d3c64b630bf8d0809914a1fff25220de2423: Status 404 returned error can't find the container with id d35de4d743d724d234350ae63e00d3c64b630bf8d0809914a1fff25220de2423
Nov 01 06:30:00.960696 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:30:00.960674    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f2f492_8bc5_4789_9a7f_c4bc9ae68e5c.slice/crio-80c2935e606687a3dcbb7617e903f1a09e0783edb34ddd611d75589900b89bd9 WatchSource:0}: Error finding container 80c2935e606687a3dcbb7617e903f1a09e0783edb34ddd611d75589900b89bd9: Status 404 returned error can't find the container with id 80c2935e606687a3dcbb7617e903f1a09e0783edb34ddd611d75589900b89bd9
Nov 01 06:30:00.962189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:00.962169    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"]
Nov 01 06:30:01.457011 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.456922    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" event={"ID":"0683ea85-a108-4759-a595-19434ec813d4","Type":"ContainerStarted","Data":"af61b8492f8bfd5c6448b28559a8b84c2c0dc922f44ba11e5f94a920490b607b"}
Nov 01 06:30:01.457011 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.456982    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" event={"ID":"0683ea85-a108-4759-a595-19434ec813d4","Type":"ContainerStarted","Data":"8d310e4730ab6ec6b6321c83654995514a7e68cbc913f7d508ff559b3de97d97"}
Nov 01 06:30:01.458526 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.458500    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" event={"ID":"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c","Type":"ContainerStarted","Data":"7aabbb1c5c48a3fdf31eadf415d2800b23b630a5a88ec48ee634fd923ebfbd1b"}
Nov 01 06:30:01.458620 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.458546    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" event={"ID":"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c","Type":"ContainerStarted","Data":"80c2935e606687a3dcbb7617e903f1a09e0783edb34ddd611d75589900b89bd9"}
Nov 01 06:30:01.459974 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.459951    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" event={"ID":"9e224c5d-28de-49b9-9d13-1ea12e4b3628","Type":"ContainerStarted","Data":"7225813660f7f72076532a06fee1c827872709c5815ca16d5d5738842f0775ce"}
Nov 01 06:30:01.460066 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.459993    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" event={"ID":"9e224c5d-28de-49b9-9d13-1ea12e4b3628","Type":"ContainerStarted","Data":"a25cc639007090eff25e74f432777bfd2a5487b0fe86827794b176e42dd21b7a"}
Nov 01 06:30:01.461439 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.461419    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" event={"ID":"453beafd-45c7-4526-895d-2bebf0f9aad4","Type":"ContainerStarted","Data":"1711b9a28e5063b5fcfa1459a25402fdb32d0a5de3b9f3b697ad65fcce985e11"}
Nov 01 06:30:01.461533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.461453    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" event={"ID":"453beafd-45c7-4526-895d-2bebf0f9aad4","Type":"ContainerStarted","Data":"d35de4d743d724d234350ae63e00d3c64b630bf8d0809914a1fff25220de2423"}
Nov 01 06:30:01.572633 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.572569    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" podStartSLOduration=1.5725469159999999 podStartE2EDuration="1.572546916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:01.570914966 +0000 UTC m=+2638.448888376" watchObservedRunningTime="2026-11-01 06:30:01.572546916 +0000 UTC m=+2638.450520327"
Nov 01 06:30:01.686988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.686938    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" podStartSLOduration=1.686925249 podStartE2EDuration="1.686925249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:01.685270637 +0000 UTC m=+2638.563244050" watchObservedRunningTime="2026-11-01 06:30:01.686925249 +0000 UTC m=+2638.564898658"
Nov 01 06:30:01.802121 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.802072    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" podStartSLOduration=1.802058398 podStartE2EDuration="1.802058398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:01.799619364 +0000 UTC m=+2638.677592775" watchObservedRunningTime="2026-11-01 06:30:01.802058398 +0000 UTC m=+2638.680031856"
Nov 01 06:30:01.914568 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:01.914517    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" podStartSLOduration=1.914502419 podStartE2EDuration="1.914502419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:01.913254239 +0000 UTC m=+2638.791227649" watchObservedRunningTime="2026-11-01 06:30:01.914502419 +0000 UTC m=+2638.792475830"
Nov 01 06:30:02.468815 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.468783    5510 generic.go:356] "Generic (PLEG): container finished" podID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerID="1711b9a28e5063b5fcfa1459a25402fdb32d0a5de3b9f3b697ad65fcce985e11" exitCode=60
Nov 01 06:30:02.469321 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.468860    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" event={"ID":"453beafd-45c7-4526-895d-2bebf0f9aad4","Type":"ContainerDied","Data":"1711b9a28e5063b5fcfa1459a25402fdb32d0a5de3b9f3b697ad65fcce985e11"}
Nov 01 06:30:02.469321 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.469216    5510 scope.go:122] "RemoveContainer" containerID="1711b9a28e5063b5fcfa1459a25402fdb32d0a5de3b9f3b697ad65fcce985e11"
Nov 01 06:30:02.470682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.470659    5510 generic.go:356] "Generic (PLEG): container finished" podID="0683ea85-a108-4759-a595-19434ec813d4" containerID="af61b8492f8bfd5c6448b28559a8b84c2c0dc922f44ba11e5f94a920490b607b" exitCode=60
Nov 01 06:30:02.470792 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.470767    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" event={"ID":"0683ea85-a108-4759-a595-19434ec813d4","Type":"ContainerDied","Data":"af61b8492f8bfd5c6448b28559a8b84c2c0dc922f44ba11e5f94a920490b607b"}
Nov 01 06:30:02.471035 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.471018    5510 scope.go:122] "RemoveContainer" containerID="af61b8492f8bfd5c6448b28559a8b84c2c0dc922f44ba11e5f94a920490b607b"
Nov 01 06:30:02.472530 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.472429    5510 generic.go:356] "Generic (PLEG): container finished" podID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerID="7aabbb1c5c48a3fdf31eadf415d2800b23b630a5a88ec48ee634fd923ebfbd1b" exitCode=60
Nov 01 06:30:02.472530 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.472505    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" event={"ID":"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c","Type":"ContainerDied","Data":"7aabbb1c5c48a3fdf31eadf415d2800b23b630a5a88ec48ee634fd923ebfbd1b"}
Nov 01 06:30:02.472707 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.472681    5510 scope.go:122] "RemoveContainer" containerID="7aabbb1c5c48a3fdf31eadf415d2800b23b630a5a88ec48ee634fd923ebfbd1b"
Nov 01 06:30:02.474228 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.474198    5510 generic.go:356] "Generic (PLEG): container finished" podID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerID="7225813660f7f72076532a06fee1c827872709c5815ca16d5d5738842f0775ce" exitCode=60
Nov 01 06:30:02.474295 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.474257    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" event={"ID":"9e224c5d-28de-49b9-9d13-1ea12e4b3628","Type":"ContainerDied","Data":"7225813660f7f72076532a06fee1c827872709c5815ca16d5d5738842f0775ce"}
Nov 01 06:30:02.474465 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:02.474450    5510 scope.go:122] "RemoveContainer" containerID="7225813660f7f72076532a06fee1c827872709c5815ca16d5d5738842f0775ce"
Nov 01 06:30:03.484241 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:03.484208    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" event={"ID":"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c","Type":"ContainerStarted","Data":"616ba301487ac8bc15c140eea0e772ae6a3f48de5a89ffe5a6acfd707cc5c9d6"}
Nov 01 06:30:03.486137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:03.486114    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" event={"ID":"9e224c5d-28de-49b9-9d13-1ea12e4b3628","Type":"ContainerStarted","Data":"a6cac27e97e6ec64b42da46b03a3f498d5f63dce4bb057c567e0ef773cb92a35"}
Nov 01 06:30:03.488149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:03.488125    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" event={"ID":"453beafd-45c7-4526-895d-2bebf0f9aad4","Type":"ContainerStarted","Data":"a0d1bd983d4cff52367299ced190183eca386bf36f8f001abdc38e6cdf103619"}
Nov 01 06:30:03.489905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:03.489874    5510 generic.go:356] "Generic (PLEG): container finished" podID="0683ea85-a108-4759-a595-19434ec813d4" containerID="ccf1614dc5001e6efb72f3962e68234fdce3666d55c0dbff9bfff14fde09c69a" exitCode=60
Nov 01 06:30:03.489995 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:03.489956    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" event={"ID":"0683ea85-a108-4759-a595-19434ec813d4","Type":"ContainerDied","Data":"ccf1614dc5001e6efb72f3962e68234fdce3666d55c0dbff9bfff14fde09c69a"}
Nov 01 06:30:03.490109 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:03.490012    5510 scope.go:122] "RemoveContainer" containerID="af61b8492f8bfd5c6448b28559a8b84c2c0dc922f44ba11e5f94a920490b607b"
Nov 01 06:30:03.490186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:03.490167    5510 scope.go:122] "RemoveContainer" containerID="ccf1614dc5001e6efb72f3962e68234fdce3666d55c0dbff9bfff14fde09c69a"
Nov 01 06:30:03.490438 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:03.490418    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db_odh-ai-gateway-infra(0683ea85-a108-4759-a595-19434ec813d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" podUID="0683ea85-a108-4759-a595-19434ec813d4"
Nov 01 06:30:04.189273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.189239    5510 scope.go:122] "RemoveContainer" containerID="daab718057c84b23593d71c7d8c2e682976081aedf28b4ad83045eb8a10b088b"
Nov 01 06:30:04.497665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.497639    5510 scope.go:122] "RemoveContainer" containerID="ccf1614dc5001e6efb72f3962e68234fdce3666d55c0dbff9bfff14fde09c69a"
Nov 01 06:30:04.498165 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:04.497967    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db_odh-ai-gateway-infra(0683ea85-a108-4759-a595-19434ec813d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" podUID="0683ea85-a108-4759-a595-19434ec813d4"
Nov 01 06:30:04.499296 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.499275    5510 generic.go:356] "Generic (PLEG): container finished" podID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerID="616ba301487ac8bc15c140eea0e772ae6a3f48de5a89ffe5a6acfd707cc5c9d6" exitCode=60
Nov 01 06:30:04.499407 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.499355    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" event={"ID":"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c","Type":"ContainerDied","Data":"616ba301487ac8bc15c140eea0e772ae6a3f48de5a89ffe5a6acfd707cc5c9d6"}
Nov 01 06:30:04.499475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.499420    5510 scope.go:122] "RemoveContainer" containerID="7aabbb1c5c48a3fdf31eadf415d2800b23b630a5a88ec48ee634fd923ebfbd1b"
Nov 01 06:30:04.499591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.499576    5510 scope.go:122] "RemoveContainer" containerID="616ba301487ac8bc15c140eea0e772ae6a3f48de5a89ffe5a6acfd707cc5c9d6"
Nov 01 06:30:04.499903 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:04.499885    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d_odh-ai-gateway-infra(f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c"
Nov 01 06:30:04.501356 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.501338    5510 generic.go:356] "Generic (PLEG): container finished" podID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerID="a6cac27e97e6ec64b42da46b03a3f498d5f63dce4bb057c567e0ef773cb92a35" exitCode=60
Nov 01 06:30:04.501457 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.501395    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" event={"ID":"9e224c5d-28de-49b9-9d13-1ea12e4b3628","Type":"ContainerDied","Data":"a6cac27e97e6ec64b42da46b03a3f498d5f63dce4bb057c567e0ef773cb92a35"}
Nov 01 06:30:04.501594 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.501570    5510 scope.go:122] "RemoveContainer" containerID="a6cac27e97e6ec64b42da46b03a3f498d5f63dce4bb057c567e0ef773cb92a35"
Nov 01 06:30:04.501806 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:04.501789    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl_odh-ai-gateway-infra(9e224c5d-28de-49b9-9d13-1ea12e4b3628)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628"
Nov 01 06:30:04.503260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.503239    5510 generic.go:356] "Generic (PLEG): container finished" podID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerID="a0d1bd983d4cff52367299ced190183eca386bf36f8f001abdc38e6cdf103619" exitCode=60
Nov 01 06:30:04.503356 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.503289    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" event={"ID":"453beafd-45c7-4526-895d-2bebf0f9aad4","Type":"ContainerDied","Data":"a0d1bd983d4cff52367299ced190183eca386bf36f8f001abdc38e6cdf103619"}
Nov 01 06:30:04.503513 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.503498    5510 scope.go:122] "RemoveContainer" containerID="a0d1bd983d4cff52367299ced190183eca386bf36f8f001abdc38e6cdf103619"
Nov 01 06:30:04.503711 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:04.503693    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd_odh-ai-gateway-infra(453beafd-45c7-4526-895d-2bebf0f9aad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4"
Nov 01 06:30:04.513634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.513617    5510 scope.go:122] "RemoveContainer" containerID="7225813660f7f72076532a06fee1c827872709c5815ca16d5d5738842f0775ce"
Nov 01 06:30:04.526072 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:04.526054    5510 scope.go:122] "RemoveContainer" containerID="1711b9a28e5063b5fcfa1459a25402fdb32d0a5de3b9f3b697ad65fcce985e11"
Nov 01 06:30:05.513536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:05.513502    5510 scope.go:122] "RemoveContainer" containerID="a0d1bd983d4cff52367299ced190183eca386bf36f8f001abdc38e6cdf103619"
Nov 01 06:30:05.514077 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:05.513805    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd_odh-ai-gateway-infra(453beafd-45c7-4526-895d-2bebf0f9aad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4"
Nov 01 06:30:05.515222 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:05.515198    5510 scope.go:122] "RemoveContainer" containerID="616ba301487ac8bc15c140eea0e772ae6a3f48de5a89ffe5a6acfd707cc5c9d6"
Nov 01 06:30:05.515504 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:05.515481    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d_odh-ai-gateway-infra(f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c"
Nov 01 06:30:05.516692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:05.516675    5510 scope.go:122] "RemoveContainer" containerID="a6cac27e97e6ec64b42da46b03a3f498d5f63dce4bb057c567e0ef773cb92a35"
Nov 01 06:30:05.516913 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:05.516895    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl_odh-ai-gateway-infra(9e224c5d-28de-49b9-9d13-1ea12e4b3628)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628"
Nov 01 06:30:16.223026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:16.222980    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:30:16.229139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:16.229116    5510 scope.go:122] "RemoveContainer" containerID="a0d1bd983d4cff52367299ced190183eca386bf36f8f001abdc38e6cdf103619"
Nov 01 06:30:16.596956 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:16.596921    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" event={"ID":"453beafd-45c7-4526-895d-2bebf0f9aad4","Type":"ContainerStarted","Data":"b6bd660d608744b442b48a5b4caedd6ff9f74e5f03519e466fc90d23cbe68410"}
Nov 01 06:30:17.229745 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:17.229655    5510 scope.go:122] "RemoveContainer" containerID="616ba301487ac8bc15c140eea0e772ae6a3f48de5a89ffe5a6acfd707cc5c9d6"
Nov 01 06:30:17.230249 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:17.229794    5510 scope.go:122] "RemoveContainer" containerID="ccf1614dc5001e6efb72f3962e68234fdce3666d55c0dbff9bfff14fde09c69a"
Nov 01 06:30:17.230249 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:17.229874    5510 scope.go:122] "RemoveContainer" containerID="a6cac27e97e6ec64b42da46b03a3f498d5f63dce4bb057c567e0ef773cb92a35"
Nov 01 06:30:17.546363 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:17.546320    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:30:17.604548 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:17.604514    5510 generic.go:356] "Generic (PLEG): container finished" podID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerID="b6bd660d608744b442b48a5b4caedd6ff9f74e5f03519e466fc90d23cbe68410" exitCode=60
Nov 01 06:30:17.604716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:17.604585    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" event={"ID":"453beafd-45c7-4526-895d-2bebf0f9aad4","Type":"ContainerDied","Data":"b6bd660d608744b442b48a5b4caedd6ff9f74e5f03519e466fc90d23cbe68410"}
Nov 01 06:30:17.604716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:17.604646    5510 scope.go:122] "RemoveContainer" containerID="a0d1bd983d4cff52367299ced190183eca386bf36f8f001abdc38e6cdf103619"
Nov 01 06:30:17.605085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:17.605069    5510 scope.go:122] "RemoveContainer" containerID="b6bd660d608744b442b48a5b4caedd6ff9f74e5f03519e466fc90d23cbe68410"
Nov 01 06:30:17.605403 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:17.605383    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd_odh-ai-gateway-infra(453beafd-45c7-4526-895d-2bebf0f9aad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4"
Nov 01 06:30:17.721627 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:17.721584    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"]
Nov 01 06:30:18.613273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.613194    5510 generic.go:356] "Generic (PLEG): container finished" podID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerID="ea3c1dc2483f8b7b287e298b2fdd1c75e790151a253252ae458807b92673b726" exitCode=60
Nov 01 06:30:18.613749 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.613268    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" event={"ID":"9e224c5d-28de-49b9-9d13-1ea12e4b3628","Type":"ContainerDied","Data":"ea3c1dc2483f8b7b287e298b2fdd1c75e790151a253252ae458807b92673b726"}
Nov 01 06:30:18.613749 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.613321    5510 scope.go:122] "RemoveContainer" containerID="a6cac27e97e6ec64b42da46b03a3f498d5f63dce4bb057c567e0ef773cb92a35"
Nov 01 06:30:18.613749 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.613688    5510 scope.go:122] "RemoveContainer" containerID="ea3c1dc2483f8b7b287e298b2fdd1c75e790151a253252ae458807b92673b726"
Nov 01 06:30:18.614041 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:18.614018    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl_odh-ai-gateway-infra(9e224c5d-28de-49b9-9d13-1ea12e4b3628)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628"
Nov 01 06:30:18.617265 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.617206    5510 generic.go:356] "Generic (PLEG): container finished" podID="0683ea85-a108-4759-a595-19434ec813d4" containerID="e897a0ba6068946c25283c50ce9104268ce227e4020d6832027e91c423c1f430" exitCode=60
Nov 01 06:30:18.617351 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.617277    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" event={"ID":"0683ea85-a108-4759-a595-19434ec813d4","Type":"ContainerDied","Data":"e897a0ba6068946c25283c50ce9104268ce227e4020d6832027e91c423c1f430"}
Nov 01 06:30:18.617691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.617650    5510 scope.go:122] "RemoveContainer" containerID="e897a0ba6068946c25283c50ce9104268ce227e4020d6832027e91c423c1f430"
Nov 01 06:30:18.618003 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:18.617977    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db_odh-ai-gateway-infra(0683ea85-a108-4759-a595-19434ec813d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" podUID="0683ea85-a108-4759-a595-19434ec813d4"
Nov 01 06:30:18.619294 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.619277    5510 generic.go:356] "Generic (PLEG): container finished" podID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerID="f89a23ea205a858608f683aa813301ef9334033a812ca91cad90c6761dd4d33c" exitCode=60
Nov 01 06:30:18.619385 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.619306    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" event={"ID":"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c","Type":"ContainerDied","Data":"f89a23ea205a858608f683aa813301ef9334033a812ca91cad90c6761dd4d33c"}
Nov 01 06:30:18.619610 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.619596    5510 scope.go:122] "RemoveContainer" containerID="f89a23ea205a858608f683aa813301ef9334033a812ca91cad90c6761dd4d33c"
Nov 01 06:30:18.619857 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:18.619817    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d_odh-ai-gateway-infra(f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c"
Nov 01 06:30:18.627666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.627643    5510 scope.go:122] "RemoveContainer" containerID="ccf1614dc5001e6efb72f3962e68234fdce3666d55c0dbff9bfff14fde09c69a"
Nov 01 06:30:18.642873 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.642852    5510 scope.go:122] "RemoveContainer" containerID="616ba301487ac8bc15c140eea0e772ae6a3f48de5a89ffe5a6acfd707cc5c9d6"
Nov 01 06:30:18.786516 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.786479    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:18.857983 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.857945    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453beafd-45c7-4526-895d-2bebf0f9aad4-kube-api-access-d8865\" (UniqueName: \"kubernetes.io/projected/453beafd-45c7-4526-895d-2bebf0f9aad4-kube-api-access-d8865\") pod \"453beafd-45c7-4526-895d-2bebf0f9aad4\" (UID: \"453beafd-45c7-4526-895d-2bebf0f9aad4\") "
Nov 01 06:30:18.858166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.858145    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453beafd-45c7-4526-895d-2bebf0f9aad4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453beafd-45c7-4526-895d-2bebf0f9aad4-maas-api-service-ca\") pod \"453beafd-45c7-4526-895d-2bebf0f9aad4\" (UID: \"453beafd-45c7-4526-895d-2bebf0f9aad4\") "
Nov 01 06:30:18.858535 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.858505    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453beafd-45c7-4526-895d-2bebf0f9aad4-maas-api-service-ca" pod "453beafd-45c7-4526-895d-2bebf0f9aad4" (UID: "453beafd-45c7-4526-895d-2bebf0f9aad4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:18.858751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.858630    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453beafd-45c7-4526-895d-2bebf0f9aad4-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:18.860296 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.860272    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453beafd-45c7-4526-895d-2bebf0f9aad4-kube-api-access-d8865" pod "453beafd-45c7-4526-895d-2bebf0f9aad4" (UID: "453beafd-45c7-4526-895d-2bebf0f9aad4"). InnerVolumeSpecName "kube-api-access-d8865". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:18.959942 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:18.959806    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8865\" (UniqueName: \"kubernetes.io/projected/453beafd-45c7-4526-895d-2bebf0f9aad4-kube-api-access-d8865\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:19.631317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.631230    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd" event={"ID":"453beafd-45c7-4526-895d-2bebf0f9aad4","Type":"ContainerDied","Data":"d35de4d743d724d234350ae63e00d3c64b630bf8d0809914a1fff25220de2423"}
Nov 01 06:30:19.631317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.631261    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"
Nov 01 06:30:19.631317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.631288    5510 scope.go:122] "RemoveContainer" containerID="b6bd660d608744b442b48a5b4caedd6ff9f74e5f03519e466fc90d23cbe68410"
Nov 01 06:30:19.741713 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.741678    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"]
Nov 01 06:30:19.800101 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.800068    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"]
Nov 01 06:30:19.803319 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.803128    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f13905-29891910-nhgdd"]
Nov 01 06:30:19.856036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.856005    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"]
Nov 01 06:30:19.942883 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.942858    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:19.970141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.970096    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e224c5d-28de-49b9-9d13-1ea12e4b3628-kube-api-access-9fpwq\" (UniqueName: \"kubernetes.io/projected/9e224c5d-28de-49b9-9d13-1ea12e4b3628-kube-api-access-9fpwq\") pod \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\" (UID: \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\") "
Nov 01 06:30:19.970327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.970190    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e224c5d-28de-49b9-9d13-1ea12e4b3628-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e224c5d-28de-49b9-9d13-1ea12e4b3628-maas-api-service-ca\") pod \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\" (UID: \"9e224c5d-28de-49b9-9d13-1ea12e4b3628\") "
Nov 01 06:30:19.970656 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.970628    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e224c5d-28de-49b9-9d13-1ea12e4b3628-maas-api-service-ca" pod "9e224c5d-28de-49b9-9d13-1ea12e4b3628" (UID: "9e224c5d-28de-49b9-9d13-1ea12e4b3628"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:19.970874 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.970849    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e224c5d-28de-49b9-9d13-1ea12e4b3628-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:19.972481 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.972455    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e224c5d-28de-49b9-9d13-1ea12e4b3628-kube-api-access-9fpwq" pod "9e224c5d-28de-49b9-9d13-1ea12e4b3628" (UID: "9e224c5d-28de-49b9-9d13-1ea12e4b3628"). InnerVolumeSpecName "kube-api-access-9fpwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:19.973007 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:19.972984    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"]
Nov 01 06:30:20.060198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.060168    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:20.072952 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.072922    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fpwq\" (UniqueName: \"kubernetes.io/projected/9e224c5d-28de-49b9-9d13-1ea12e4b3628-kube-api-access-9fpwq\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:20.113945 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.113920    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:20.173933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.173848    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-maas-api-service-ca\") pod \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\" (UID: \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\") "
Nov 01 06:30:20.173933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.173893    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0683ea85-a108-4759-a595-19434ec813d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683ea85-a108-4759-a595-19434ec813d4-maas-api-service-ca\") pod \"0683ea85-a108-4759-a595-19434ec813d4\" (UID: \"0683ea85-a108-4759-a595-19434ec813d4\") "
Nov 01 06:30:20.174159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.174071    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0683ea85-a108-4759-a595-19434ec813d4-kube-api-access-bxs4w\" (UniqueName: \"kubernetes.io/projected/0683ea85-a108-4759-a595-19434ec813d4-kube-api-access-bxs4w\") pod \"0683ea85-a108-4759-a595-19434ec813d4\" (UID: \"0683ea85-a108-4759-a595-19434ec813d4\") "
Nov 01 06:30:20.174159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.174122    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-kube-api-access-dblhg\" (UniqueName: \"kubernetes.io/projected/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-kube-api-access-dblhg\") pod \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\" (UID: \"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c\") "
Nov 01 06:30:20.174396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.174319    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0683ea85-a108-4759-a595-19434ec813d4-maas-api-service-ca" pod "0683ea85-a108-4759-a595-19434ec813d4" (UID: "0683ea85-a108-4759-a595-19434ec813d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:20.174396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.174344    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-maas-api-service-ca" pod "f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" (UID: "f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:20.174612 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.174576    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:20.174612 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.174600    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683ea85-a108-4759-a595-19434ec813d4-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:20.176968 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.176946    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-kube-api-access-dblhg" pod "f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" (UID: "f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c"). InnerVolumeSpecName "kube-api-access-dblhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:20.177036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.176987    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0683ea85-a108-4759-a595-19434ec813d4-kube-api-access-bxs4w" pod "0683ea85-a108-4759-a595-19434ec813d4" (UID: "0683ea85-a108-4759-a595-19434ec813d4"). InnerVolumeSpecName "kube-api-access-bxs4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:20.275535 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.275503    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxs4w\" (UniqueName: \"kubernetes.io/projected/0683ea85-a108-4759-a595-19434ec813d4-kube-api-access-bxs4w\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:20.275535 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.275528    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dblhg\" (UniqueName: \"kubernetes.io/projected/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c-kube-api-access-dblhg\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:20.639545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.639518    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"
Nov 01 06:30:20.639545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.639535    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d" event={"ID":"f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c","Type":"ContainerDied","Data":"80c2935e606687a3dcbb7617e903f1a09e0783edb34ddd611d75589900b89bd9"}
Nov 01 06:30:20.640117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.639593    5510 scope.go:122] "RemoveContainer" containerID="f89a23ea205a858608f683aa813301ef9334033a812ca91cad90c6761dd4d33c"
Nov 01 06:30:20.641190 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.641164    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"
Nov 01 06:30:20.641308 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.641200    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl" event={"ID":"9e224c5d-28de-49b9-9d13-1ea12e4b3628","Type":"ContainerDied","Data":"a25cc639007090eff25e74f432777bfd2a5487b0fe86827794b176e42dd21b7a"}
Nov 01 06:30:20.648542 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.648516    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db" event={"ID":"0683ea85-a108-4759-a595-19434ec813d4","Type":"ContainerDied","Data":"8d310e4730ab6ec6b6321c83654995514a7e68cbc913f7d508ff559b3de97d97"}
Nov 01 06:30:20.648655 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.648556    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"
Nov 01 06:30:20.652639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.652622    5510 scope.go:122] "RemoveContainer" containerID="ea3c1dc2483f8b7b287e298b2fdd1c75e790151a253252ae458807b92673b726"
Nov 01 06:30:20.685022 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.663132    5510 scope.go:122] "RemoveContainer" containerID="e897a0ba6068946c25283c50ce9104268ce227e4020d6832027e91c423c1f430"
Nov 01 06:30:20.806686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.806651    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"]
Nov 01 06:30:20.810566 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.810538    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9c8fc5-29891910-6k2db"]
Nov 01 06:30:20.919666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.919588    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"]
Nov 01 06:30:20.928738 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:20.928708    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cfe368-29891910-pnwfl"]
Nov 01 06:30:21.039217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:21.039187    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"]
Nov 01 06:30:21.041626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:21.041602    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c0662f25-29891910-6gd9d"]
Nov 01 06:30:21.236736 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:21.236701    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0683ea85-a108-4759-a595-19434ec813d4" path="/var/lib/kubelet/pods/0683ea85-a108-4759-a595-19434ec813d4/volumes"
Nov 01 06:30:21.237159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:21.237145    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" path="/var/lib/kubelet/pods/453beafd-45c7-4526-895d-2bebf0f9aad4/volumes"
Nov 01 06:30:21.237498 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:21.237487    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" path="/var/lib/kubelet/pods/9e224c5d-28de-49b9-9d13-1ea12e4b3628/volumes"
Nov 01 06:30:21.238161 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:21.238146    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" path="/var/lib/kubelet/pods/f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c/volumes"
Nov 01 06:30:47.588314 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:30:47.588272    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:30:54.744614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:30:54.744581    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:31:05.995657 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:05.995603    5510 generic.go:356] "Generic (PLEG): container finished" podID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerID="69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784" exitCode=0
Nov 01 06:31:05.996581 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:05.995663    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" event={"ID":"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5","Type":"ContainerDied","Data":"69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784"}
Nov 01 06:31:05.996581 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:05.996246    5510 scope.go:122] "RemoveContainer" containerID="69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784"
Nov 01 06:31:05.998888 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:05.998807    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:31:05.999035 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:05.998923    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:31:05.999035 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:05.998975    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:31:06.257515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:06.257445    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:31:07.004893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:07.004852    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" event={"ID":"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5","Type":"ContainerStarted","Data":"5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689"}
Nov 01 06:31:07.227220 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:07.227185    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:31:07.228160 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:07.228132    5510 patch_prober.go:29] interesting pod/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 01 06:31:07.228311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:07.228178    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 01 06:31:08.227358 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:08.227324    5510 patch_prober.go:29] interesting pod/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 01 06:31:08.227814 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:08.227375    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 01 06:31:09.227081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:09.227041    5510 patch_prober.go:29] interesting pod/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 01 06:31:09.227267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:09.227099    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 01 06:31:10.227599 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:10.227556    5510 patch_prober.go:29] interesting pod/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 01 06:31:10.228135 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:10.227617    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 01 06:31:11.226798 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:11.226765    5510 patch_prober.go:29] interesting pod/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 01 06:31:11.227011 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:11.226816    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 01 06:31:12.227299 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:12.227261    5510 patch_prober.go:29] interesting pod/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 01 06:31:12.227781 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:12.227323    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 01 06:31:13.236042 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:13.236009    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:31:13.236535 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:13.236372    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:31:13.236995 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:13.236977    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:31:17.552309 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:31:17.552267    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:31:17.826510 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:17.826432    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:31:47.544269 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:31:47.544232    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:31:49.827355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:49.827322    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"]
Nov 01 06:31:49.932168 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:49.932137    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"]
Nov 01 06:31:54.827646 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:54.827604    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj" podUID="533f1b8d-f031-4281-a031-6aa330912c57" containerName="payload-processing" containerID="cri-o://576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236" gracePeriod=25
Nov 01 06:31:54.933441 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:54.933380    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z" podUID="7fbed767-72b7-4db9-aad3-e778d95f5f4d" containerName="payload-pre-processing" containerID="cri-o://32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc" gracePeriod=25
Nov 01 06:31:55.087531 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.087465    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:31:55.189195 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.189168    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533f1b8d-f031-4281-a031-6aa330912c57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533f1b8d-f031-4281-a031-6aa330912c57-plugins-config-volume\") pod \"533f1b8d-f031-4281-a031-6aa330912c57\" (UID: \"533f1b8d-f031-4281-a031-6aa330912c57\") "
Nov 01 06:31:55.189415 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.189387    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533f1b8d-f031-4281-a031-6aa330912c57-kube-api-access-lzfmf\" (UniqueName: \"kubernetes.io/projected/533f1b8d-f031-4281-a031-6aa330912c57-kube-api-access-lzfmf\") pod \"533f1b8d-f031-4281-a031-6aa330912c57\" (UID: \"533f1b8d-f031-4281-a031-6aa330912c57\") "
Nov 01 06:31:55.189575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.189555    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533f1b8d-f031-4281-a031-6aa330912c57-plugins-config-volume" pod "533f1b8d-f031-4281-a031-6aa330912c57" (UID: "533f1b8d-f031-4281-a031-6aa330912c57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:31:55.189818 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.189801    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533f1b8d-f031-4281-a031-6aa330912c57-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:55.191447 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.191427    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533f1b8d-f031-4281-a031-6aa330912c57-kube-api-access-lzfmf" pod "533f1b8d-f031-4281-a031-6aa330912c57" (UID: "533f1b8d-f031-4281-a031-6aa330912c57"). InnerVolumeSpecName "kube-api-access-lzfmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:55.204403 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.204377    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:31:55.291244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.291217    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fbed767-72b7-4db9-aad3-e778d95f5f4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbed767-72b7-4db9-aad3-e778d95f5f4d-plugins-config-volume\") pod \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\" (UID: \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\") "
Nov 01 06:31:55.291426 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.291308    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fbed767-72b7-4db9-aad3-e778d95f5f4d-kube-api-access-ftgx9\" (UniqueName: \"kubernetes.io/projected/7fbed767-72b7-4db9-aad3-e778d95f5f4d-kube-api-access-ftgx9\") pod \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\" (UID: \"7fbed767-72b7-4db9-aad3-e778d95f5f4d\") "
Nov 01 06:31:55.291575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.291553    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzfmf\" (UniqueName: \"kubernetes.io/projected/533f1b8d-f031-4281-a031-6aa330912c57-kube-api-access-lzfmf\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:55.291645 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.291624    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbed767-72b7-4db9-aad3-e778d95f5f4d-plugins-config-volume" pod "7fbed767-72b7-4db9-aad3-e778d95f5f4d" (UID: "7fbed767-72b7-4db9-aad3-e778d95f5f4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:31:55.293578 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.293556    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbed767-72b7-4db9-aad3-e778d95f5f4d-kube-api-access-ftgx9" pod "7fbed767-72b7-4db9-aad3-e778d95f5f4d" (UID: "7fbed767-72b7-4db9-aad3-e778d95f5f4d"). InnerVolumeSpecName "kube-api-access-ftgx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:55.333449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.333416    5510 generic.go:356] "Generic (PLEG): container finished" podID="7fbed767-72b7-4db9-aad3-e778d95f5f4d" containerID="32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc" exitCode=0
Nov 01 06:31:55.333614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.333527    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"
Nov 01 06:31:55.333614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.333551    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z" event={"ID":"7fbed767-72b7-4db9-aad3-e778d95f5f4d","Type":"ContainerDied","Data":"32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc"}
Nov 01 06:31:55.333724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.333619    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z" event={"ID":"7fbed767-72b7-4db9-aad3-e778d95f5f4d","Type":"ContainerDied","Data":"19bba0bf9c3b08eb592c3a40ee5d84f46025a7c8fd611c5fd76ce3232edaa2bd"}
Nov 01 06:31:55.333724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.333644    5510 scope.go:122] "RemoveContainer" containerID="32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc"
Nov 01 06:31:55.336436 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.336411    5510 generic.go:356] "Generic (PLEG): container finished" podID="533f1b8d-f031-4281-a031-6aa330912c57" containerID="576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236" exitCode=0
Nov 01 06:31:55.336575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.336521    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj" event={"ID":"533f1b8d-f031-4281-a031-6aa330912c57","Type":"ContainerDied","Data":"576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236"}
Nov 01 06:31:55.336575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.336561    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj" event={"ID":"533f1b8d-f031-4281-a031-6aa330912c57","Type":"ContainerDied","Data":"c24be151ededca8924d9d4dde652d080cfc0a3021d394ef7c4e081e802600415"}
Nov 01 06:31:55.336704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.336629    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"
Nov 01 06:31:55.351148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.351126    5510 scope.go:122] "RemoveContainer" containerID="32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc"
Nov 01 06:31:55.351560 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:31:55.351536    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc\": container with ID starting with 32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc not found: ID does not exist" containerID="32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc"
Nov 01 06:31:55.351677 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.351569    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc"} err="failed to get container status \"32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc\": rpc error: code = NotFound desc = could not find container \"32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc\": container with ID starting with 32bddeaba98dbbdc45cbddbf719d317c2ce021815fbaa7c83cc51938901871cc not found: ID does not exist"
Nov 01 06:31:55.351677 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.351593    5510 scope.go:122] "RemoveContainer" containerID="576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236"
Nov 01 06:31:55.362191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.362167    5510 scope.go:122] "RemoveContainer" containerID="576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236"
Nov 01 06:31:55.362491 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:31:55.362464    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236\": container with ID starting with 576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236 not found: ID does not exist" containerID="576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236"
Nov 01 06:31:55.362569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.362502    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236"} err="failed to get container status \"576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236\": rpc error: code = NotFound desc = could not find container \"576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236\": container with ID starting with 576b87b9e1f98ac8a1e439d725e750fee618c6bf81ea2e442d52d11b704a0236 not found: ID does not exist"
Nov 01 06:31:55.392795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.392757    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbed767-72b7-4db9-aad3-e778d95f5f4d-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:55.392795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.392781    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgx9\" (UniqueName: \"kubernetes.io/projected/7fbed767-72b7-4db9-aad3-e778d95f5f4d-kube-api-access-ftgx9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:55.497544 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.497515    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"]
Nov 01 06:31:55.499730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.499710    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-745a18-7d76b7c4cf-nd67z"]
Nov 01 06:31:55.607468 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.607392    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"]
Nov 01 06:31:55.610890 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.610868    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-745a18-b6c7bb749-hfzsj"]
Nov 01 06:31:55.903057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:55.902985    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"]
Nov 01 06:31:57.236536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:57.236505    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533f1b8d-f031-4281-a031-6aa330912c57" path="/var/lib/kubelet/pods/533f1b8d-f031-4281-a031-6aa330912c57/volumes"
Nov 01 06:31:57.236946 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:57.236910    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fbed767-72b7-4db9-aad3-e778d95f5f4d" path="/var/lib/kubelet/pods/7fbed767-72b7-4db9-aad3-e778d95f5f4d/volumes"
Nov 01 06:31:58.621755 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.621724    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"]
Nov 01 06:31:58.622255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.622047    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk" podUID="d0891aff-10dc-4366-87dc-5e1e3afe935f" containerName="maas-api" containerID="cri-o://77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15" gracePeriod=30
Nov 01 06:31:58.915421 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.915400    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:31:58.975988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.975964    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0891aff-10dc-4366-87dc-5e1e3afe935f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0891aff-10dc-4366-87dc-5e1e3afe935f-openshift-service-ca\") pod \"d0891aff-10dc-4366-87dc-5e1e3afe935f\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") "
Nov 01 06:31:58.976092 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.976004    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0891aff-10dc-4366-87dc-5e1e3afe935f-kube-api-access-ngzfm\" (UniqueName: \"kubernetes.io/projected/d0891aff-10dc-4366-87dc-5e1e3afe935f-kube-api-access-ngzfm\") pod \"d0891aff-10dc-4366-87dc-5e1e3afe935f\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") "
Nov 01 06:31:58.976092 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.976042    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-metrics-certs\") pod \"d0891aff-10dc-4366-87dc-5e1e3afe935f\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") "
Nov 01 06:31:58.976303 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.976282    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-maas-api-tls\") pod \"d0891aff-10dc-4366-87dc-5e1e3afe935f\" (UID: \"d0891aff-10dc-4366-87dc-5e1e3afe935f\") "
Nov 01 06:31:58.976364 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.976332    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0891aff-10dc-4366-87dc-5e1e3afe935f-openshift-service-ca" pod "d0891aff-10dc-4366-87dc-5e1e3afe935f" (UID: "d0891aff-10dc-4366-87dc-5e1e3afe935f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:31:58.976680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.976658    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0891aff-10dc-4366-87dc-5e1e3afe935f-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:58.978357 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.978334    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-maas-api-tls" pod "d0891aff-10dc-4366-87dc-5e1e3afe935f" (UID: "d0891aff-10dc-4366-87dc-5e1e3afe935f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:31:58.978461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.978361    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-metrics-certs" pod "d0891aff-10dc-4366-87dc-5e1e3afe935f" (UID: "d0891aff-10dc-4366-87dc-5e1e3afe935f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:31:58.978461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:58.978387    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0891aff-10dc-4366-87dc-5e1e3afe935f-kube-api-access-ngzfm" pod "d0891aff-10dc-4366-87dc-5e1e3afe935f" (UID: "d0891aff-10dc-4366-87dc-5e1e3afe935f"). InnerVolumeSpecName "kube-api-access-ngzfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:59.077540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.077506    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzfm\" (UniqueName: \"kubernetes.io/projected/d0891aff-10dc-4366-87dc-5e1e3afe935f-kube-api-access-ngzfm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:59.077540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.077544    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:59.077751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.077559    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0891aff-10dc-4366-87dc-5e1e3afe935f-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:59.463518 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.463483    5510 generic.go:356] "Generic (PLEG): container finished" podID="d0891aff-10dc-4366-87dc-5e1e3afe935f" containerID="77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15" exitCode=0
Nov 01 06:31:59.463685 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.463524    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"
Nov 01 06:31:59.463685 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.463566    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk" event={"ID":"d0891aff-10dc-4366-87dc-5e1e3afe935f","Type":"ContainerDied","Data":"77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15"}
Nov 01 06:31:59.463685 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.463621    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk" event={"ID":"d0891aff-10dc-4366-87dc-5e1e3afe935f","Type":"ContainerDied","Data":"3f9ca620acff69def3bf4054a5152668339a69a09222d961d6916a6d5c90797c"}
Nov 01 06:31:59.463685 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.463648    5510 scope.go:122] "RemoveContainer" containerID="77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15"
Nov 01 06:31:59.473313 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.473293    5510 scope.go:122] "RemoveContainer" containerID="77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15"
Nov 01 06:31:59.473602 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:31:59.473575    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15\": container with ID starting with 77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15 not found: ID does not exist" containerID="77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15"
Nov 01 06:31:59.473661 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.473611    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15"} err="failed to get container status \"77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15\": rpc error: code = NotFound desc = could not find container \"77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15\": container with ID starting with 77c6c0e2d337873df9d1b69c54799c4fbc88cbe79aa2aa482e007721b28e0c15 not found: ID does not exist"
Nov 01 06:31:59.630159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.630067    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"]
Nov 01 06:31:59.633495 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.633469    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-745a18-64cd4749c8-wcgnk"]
Nov 01 06:31:59.970440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:31:59.970403    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:32:00.903943 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:00.903900    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb" podUID="591cae75-2f5a-4c11-8ea6-1cfc931e3eec" containerName="payload-processing" containerID="cri-o://9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d" gracePeriod=25
Nov 01 06:32:01.160001 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.159945    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:32:01.222586 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.222555    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-kube-api-access-zm2v9\" (UniqueName: \"kubernetes.io/projected/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-kube-api-access-zm2v9\") pod \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\" (UID: \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\") "
Nov 01 06:32:01.222764 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.222651    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-plugins-config-volume\") pod \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\" (UID: \"591cae75-2f5a-4c11-8ea6-1cfc931e3eec\") "
Nov 01 06:32:01.223077 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.223007    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-plugins-config-volume" pod "591cae75-2f5a-4c11-8ea6-1cfc931e3eec" (UID: "591cae75-2f5a-4c11-8ea6-1cfc931e3eec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:32:01.224699 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.224674    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-kube-api-access-zm2v9" pod "591cae75-2f5a-4c11-8ea6-1cfc931e3eec" (UID: "591cae75-2f5a-4c11-8ea6-1cfc931e3eec"). InnerVolumeSpecName "kube-api-access-zm2v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:32:01.237133 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.237111    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0891aff-10dc-4366-87dc-5e1e3afe935f" path="/var/lib/kubelet/pods/d0891aff-10dc-4366-87dc-5e1e3afe935f/volumes"
Nov 01 06:32:01.323984 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.323955    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:01.323984 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.323984    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2v9\" (UniqueName: \"kubernetes.io/projected/591cae75-2f5a-4c11-8ea6-1cfc931e3eec-kube-api-access-zm2v9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:01.478394 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.478362    5510 generic.go:356] "Generic (PLEG): container finished" podID="591cae75-2f5a-4c11-8ea6-1cfc931e3eec" containerID="9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d" exitCode=0
Nov 01 06:32:01.478579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.478404    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"
Nov 01 06:32:01.478579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.478410    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb" event={"ID":"591cae75-2f5a-4c11-8ea6-1cfc931e3eec","Type":"ContainerDied","Data":"9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d"}
Nov 01 06:32:01.478579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.478454    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb" event={"ID":"591cae75-2f5a-4c11-8ea6-1cfc931e3eec","Type":"ContainerDied","Data":"55c27a5ae4a1fb944e71ea2c3fd8618816088b22d57b97ae5c1460e1ddbc436c"}
Nov 01 06:32:01.478579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.478476    5510 scope.go:122] "RemoveContainer" containerID="9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d"
Nov 01 06:32:01.488185 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.488168    5510 scope.go:122] "RemoveContainer" containerID="9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d"
Nov 01 06:32:01.488464 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:32:01.488447    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d\": container with ID starting with 9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d not found: ID does not exist" containerID="9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d"
Nov 01 06:32:01.488545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.488472    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d"} err="failed to get container status \"9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d\": rpc error: code = NotFound desc = could not find container \"9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d\": container with ID starting with 9416442751d195a70315f478ee7b3f70d13c873799c97bf3a88194d6d00c725d not found: ID does not exist"
Nov 01 06:32:01.642165 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.642135    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"]
Nov 01 06:32:01.645813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:01.645790    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aed62e2-6445968665-qjmvb"]
Nov 01 06:32:03.235410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:03.235380    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591cae75-2f5a-4c11-8ea6-1cfc931e3eec" path="/var/lib/kubelet/pods/591cae75-2f5a-4c11-8ea6-1cfc931e3eec/volumes"
Nov 01 06:32:10.113478 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.113439    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"]
Nov 01 06:32:10.114122 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114106    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d0891aff-10dc-4366-87dc-5e1e3afe935f" containerName="maas-api"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114123    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0891aff-10dc-4366-87dc-5e1e3afe935f" containerName="maas-api"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114149    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114155    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114161    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114166    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114176    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114180    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114191    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:32:10.114194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114196    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114205    5510 cpu_manager.go:414] "containerMap: removing container" podUID="533f1b8d-f031-4281-a031-6aa330912c57" containerName="payload-processing"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114213    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533f1b8d-f031-4281-a031-6aa330912c57" containerName="payload-processing"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114226    5510 cpu_manager.go:414] "containerMap: removing container" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114231    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114249    5510 cpu_manager.go:414] "containerMap: removing container" podUID="591cae75-2f5a-4c11-8ea6-1cfc931e3eec" containerName="payload-processing"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114253    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591cae75-2f5a-4c11-8ea6-1cfc931e3eec" containerName="payload-processing"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114259    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114263    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114269    5510 cpu_manager.go:414] "containerMap: removing container" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114272    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114287    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114294    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114306    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7fbed767-72b7-4db9-aad3-e778d95f5f4d" containerName="payload-pre-processing"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114310    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fbed767-72b7-4db9-aad3-e778d95f5f4d" containerName="payload-pre-processing"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114393    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114407    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0891aff-10dc-4366-87dc-5e1e3afe935f" containerName="maas-api"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114414    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114420    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114427    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114432    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114438    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="533f1b8d-f031-4281-a031-6aa330912c57" containerName="payload-processing"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114445    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114450    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114457    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="591cae75-2f5a-4c11-8ea6-1cfc931e3eec" containerName="payload-processing"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114462    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114469    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:32:10.114606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.114474    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fbed767-72b7-4db9-aad3-e778d95f5f4d" containerName="payload-pre-processing"
Nov 01 06:32:10.117656 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.117639    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.172318 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.172271    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-747b7b-serving-cert\""
Nov 01 06:32:10.224266 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.224237    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"]
Nov 01 06:32:10.232058 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.232038    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsc7m\" (UniqueName: \"kubernetes.io/projected/5364423f-7955-4f92-afb8-f4a54ccd9c0d-kube-api-access-wsc7m\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.232155 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.232084    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.232211 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.232155    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.232211 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.232201    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5364423f-7955-4f92-afb8-f4a54ccd9c0d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.333365 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.333329    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5364423f-7955-4f92-afb8-f4a54ccd9c0d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.333655 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.333630    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsc7m\" (UniqueName: \"kubernetes.io/projected/5364423f-7955-4f92-afb8-f4a54ccd9c0d-kube-api-access-wsc7m\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.334026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.333733    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.334026 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:32:10.333812    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-serving-cert: secret "maas-api-e2e-shared-b-w1-747b7b-serving-cert" not found
Nov 01 06:32:10.334026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.333819    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.334026 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:32:10.333921    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls podName:5364423f-7955-4f92-afb8-f4a54ccd9c0d nodeName:}" failed. No retries permitted until 2026-11-01 06:32:10.833897408 +0000 UTC m=+2767.711870816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls") pod "maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk" (UID: "5364423f-7955-4f92-afb8-f4a54ccd9c0d") : secret "maas-api-e2e-shared-b-w1-747b7b-serving-cert" not found
Nov 01 06:32:10.334252 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.334080    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5364423f-7955-4f92-afb8-f4a54ccd9c0d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.336479 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.336460    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.404238 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.404161    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsc7m\" (UniqueName: \"kubernetes.io/projected/5364423f-7955-4f92-afb8-f4a54ccd9c0d-kube-api-access-wsc7m\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.845393 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.845359    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:10.847958 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:10.847935    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:11.029322 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:11.029289    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:11.409540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:11.409505    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"]
Nov 01 06:32:11.411139 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:32:11.411103    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5364423f_7955_4f92_afb8_f4a54ccd9c0d.slice/crio-a88181b6f1c10aac78bbb4f1dfd0f0ac2cd9a003d166c855e86a35996ddcb11b WatchSource:0}: Error finding container a88181b6f1c10aac78bbb4f1dfd0f0ac2cd9a003d166c855e86a35996ddcb11b: Status 404 returned error can't find the container with id a88181b6f1c10aac78bbb4f1dfd0f0ac2cd9a003d166c855e86a35996ddcb11b
Nov 01 06:32:11.556889 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:11.556852    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk" event={"ID":"5364423f-7955-4f92-afb8-f4a54ccd9c0d","Type":"ContainerStarted","Data":"a88181b6f1c10aac78bbb4f1dfd0f0ac2cd9a003d166c855e86a35996ddcb11b"}
Nov 01 06:32:12.564552 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:12.564455    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk" event={"ID":"5364423f-7955-4f92-afb8-f4a54ccd9c0d","Type":"ContainerStarted","Data":"6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927"}
Nov 01 06:32:12.564552 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:12.564531    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:12.682885 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:12.682812    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk" podStartSLOduration=1.7857346349999998 podStartE2EDuration="2.682799246s" totalImagesPullingTime="897.064611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:12.680609318 +0000 UTC m=+2769.558582728" watchObservedRunningTime="2026-11-01 06:32:12.682799246 +0000 UTC m=+2769.560772636"
Nov 01 06:32:17.544965 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:32:17.544928    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:32:18.575419 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:18.575388    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:32:46.648200 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:32:46.648120    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:32:47.545810 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:32:47.545770    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:33:05.284287 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:05.284256    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"]
Nov 01 06:33:05.284755 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:05.284460    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy" containerID="cri-o://5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689" gracePeriod=30
Nov 01 06:33:10.545354 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.545328    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:33:10.608899 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.608800    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istiod-ca-cert\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.608927    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-certs\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.608952    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-token\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.608989    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-socket\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609076 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609044    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-podinfo\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609296 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609084    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-credential-socket\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609296 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609126    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-kube-api-access-22g6q\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-kube-api-access-22g6q\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609296 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609160    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-data\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609296 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609221    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-envoy\") pod \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\" (UID: \"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5\") "
Nov 01 06:33:10.609296 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609255    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-certs" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:10.609553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609305    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istiod-ca-cert" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:10.609553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609411    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-socket" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:10.609553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609513    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-credential-socket" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:10.609553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609541    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-data" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:10.609789 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609766    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.609789 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609786    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.610099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609799    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.610099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609817    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.610099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.609862    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.611523 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.611495    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-token" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:10.611649 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.611636    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-podinfo" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:33:10.611723 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.611647    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-kube-api-access-22g6q" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "kube-api-access-22g6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:10.611723 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.611687    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-envoy" pod "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" (UID: "5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:10.711209 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.711171    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22g6q\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-kube-api-access-22g6q\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.711392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.711204    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.711392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.711253    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.711392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.711262    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:10.912471 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.912382    5510 generic.go:356] "Generic (PLEG): container finished" podID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerID="5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689" exitCode=0
Nov 01 06:33:10.912471 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.912428    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"
Nov 01 06:33:10.912471 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.912460    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" event={"ID":"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5","Type":"ContainerDied","Data":"5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689"}
Nov 01 06:33:10.912725 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.912518    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh" event={"ID":"5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5","Type":"ContainerDied","Data":"b74d9b13e9bdd5b3ee9944bea883f773533130551fdf02102bc11b7e545ae99a"}
Nov 01 06:33:10.912725 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.912541    5510 scope.go:122] "RemoveContainer" containerID="5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689"
Nov 01 06:33:10.924050 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.924033    5510 scope.go:122] "RemoveContainer" containerID="69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784"
Nov 01 06:33:10.934140 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.934115    5510 scope.go:122] "RemoveContainer" containerID="5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689"
Nov 01 06:33:10.934465 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:33:10.934445    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689\": container with ID starting with 5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689 not found: ID does not exist" containerID="5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689"
Nov 01 06:33:10.934533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.934475    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689"} err="failed to get container status \"5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689\": rpc error: code = NotFound desc = could not find container \"5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689\": container with ID starting with 5b408661a15bfef06486ffe014c511e2e8123f95cdb49d0b949b39969f9d6689 not found: ID does not exist"
Nov 01 06:33:10.934533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.934494    5510 scope.go:122] "RemoveContainer" containerID="69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784"
Nov 01 06:33:10.934748 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:33:10.934729    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784\": container with ID starting with 69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784 not found: ID does not exist" containerID="69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784"
Nov 01 06:33:10.934809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:10.934761    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784"} err="failed to get container status \"69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784\": rpc error: code = NotFound desc = could not find container \"69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784\": container with ID starting with 69b6f62f4b8fca7063403b0bef0d443f78508971117aaec77fa1fbe414de5784 not found: ID does not exist"
Nov 01 06:33:11.077593 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:11.077553    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"]
Nov 01 06:33:11.083048 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:11.083022    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0aed62e2-openshift-default-675d876d77-rlqnh"]
Nov 01 06:33:11.235780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:11.235747    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" path="/var/lib/kubelet/pods/5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5/volumes"
Nov 01 06:33:17.545631 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:33:17.545586    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:33:17.588447 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:17.588406    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:33:47.546325 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:33:47.546284    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:33:53.713031 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:33:53.712997    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:34:05.912728 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.912692    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"]
Nov 01 06:34:05.913267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913254    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy"
Nov 01 06:34:05.913311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913269    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy"
Nov 01 06:34:05.913311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913294    5510 cpu_manager.go:414] "containerMap: removing container" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:34:05.913311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913300    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:34:05.913402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913315    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:34:05.913402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913320    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:34:05.913402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913343    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:34:05.913402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913347    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:34:05.913402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913355    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:34:05.913402 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913360    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0683ea85-a108-4759-a595-19434ec813d4" containerName="cleanup"
Nov 01 06:34:05.913581 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913435    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="453beafd-45c7-4526-895d-2bebf0f9aad4" containerName="cleanup"
Nov 01 06:34:05.913581 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913445    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f2f492-8bc5-4789-9a7f-c4bc9ae68e5c" containerName="cleanup"
Nov 01 06:34:05.913581 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913453    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy"
Nov 01 06:34:05.913581 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.913458    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e224c5d-28de-49b9-9d13-1ea12e4b3628" containerName="cleanup"
Nov 01 06:34:05.917797 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.917781    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:05.969744 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.969720    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9f8bf4cbae3d\""
Nov 01 06:34:05.986097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.986070    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5t5r\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-kube-api-access-x5t5r\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:05.986243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.986106    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-provider-credentials\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:05.986243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.986148    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-routing-overlay-volume\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:05.986365 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:05.986349    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-plugins-config-volume\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.019327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.019299    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 06:34:06.019512 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.019321    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 06:34:06.019512 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.019320    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qvhf2\""
Nov 01 06:34:06.019688 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.019349    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 06:34:06.019688 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.019373    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 06:34:06.023010 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.022989    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"]
Nov 01 06:34:06.087209 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.087183    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-provider-credentials\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.087338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.087243    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-routing-overlay-volume\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.087383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.087373    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-plugins-config-volume\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.087420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.087401    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5t5r\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-kube-api-access-x5t5r\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.088129 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.088100    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-routing-overlay-volume\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.088231 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.088203    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-plugins-config-volume\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.089804 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.089783    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-provider-credentials\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.147733 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.147710    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5t5r\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-kube-api-access-x5t5r\") pod \"payload-processing-external-model-6f658c5c64-p6rwt\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") " pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.231599 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.231576    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:06.609752 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:06.608060    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"]
Nov 01 06:34:06.611808 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:34:06.611777    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75aa37b_8278_4dcc_8dbd_0911807a3791.slice/crio-4ce38d0fc637d70a1f18c34274418cce706994acfce6ee64594736d6800cb87b WatchSource:0}: Error finding container 4ce38d0fc637d70a1f18c34274418cce706994acfce6ee64594736d6800cb87b: Status 404 returned error can't find the container with id 4ce38d0fc637d70a1f18c34274418cce706994acfce6ee64594736d6800cb87b
Nov 01 06:34:07.320114 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:07.320068    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt" event={"ID":"c75aa37b-8278-4dcc-8dbd-0911807a3791","Type":"ContainerStarted","Data":"31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47"}
Nov 01 06:34:07.320532 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:07.320130    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt" event={"ID":"c75aa37b-8278-4dcc-8dbd-0911807a3791","Type":"ContainerStarted","Data":"4ce38d0fc637d70a1f18c34274418cce706994acfce6ee64594736d6800cb87b"}
Nov 01 06:34:07.320532 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:07.320158    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:07.438053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:07.438001    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt" podStartSLOduration=2.4379835610000002 podStartE2EDuration="2.437983561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:07.43733428 +0000 UTC m=+2884.315307690" watchObservedRunningTime="2026-11-01 06:34:07.437983561 +0000 UTC m=+2884.315956974"
Nov 01 06:34:17.545003 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:34:17.544924    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:34:18.330346 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:18.330311    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt" podUID="c75aa37b-8278-4dcc-8dbd-0911807a3791" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:34:22.067999 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:22.067958    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-2t54h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:34:22.068423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:22.068021    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:34:27.628741 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:27.628708    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"]
Nov 01 06:34:27.630160 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:27.630136    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt" podUID="c75aa37b-8278-4dcc-8dbd-0911807a3791" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:34:32.629916 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.629871    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt" podUID="c75aa37b-8278-4dcc-8dbd-0911807a3791" containerName="payload-processing" containerID="cri-o://31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47" gracePeriod=25
Nov 01 06:34:32.899224 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.899172    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:32.974914 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.974889    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-plugins-config-volume\") pod \"c75aa37b-8278-4dcc-8dbd-0911807a3791\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") "
Nov 01 06:34:32.975139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.975119    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-routing-overlay-volume\") pod \"c75aa37b-8278-4dcc-8dbd-0911807a3791\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") "
Nov 01 06:34:32.975219 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.975196    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-kube-api-access-x5t5r\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-kube-api-access-x5t5r\") pod \"c75aa37b-8278-4dcc-8dbd-0911807a3791\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") "
Nov 01 06:34:32.975262 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.975237    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-plugins-config-volume" pod "c75aa37b-8278-4dcc-8dbd-0911807a3791" (UID: "c75aa37b-8278-4dcc-8dbd-0911807a3791"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:34:32.975262 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.975256    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-provider-credentials\") pod \"c75aa37b-8278-4dcc-8dbd-0911807a3791\" (UID: \"c75aa37b-8278-4dcc-8dbd-0911807a3791\") "
Nov 01 06:34:32.975440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.975422    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-routing-overlay-volume" pod "c75aa37b-8278-4dcc-8dbd-0911807a3791" (UID: "c75aa37b-8278-4dcc-8dbd-0911807a3791"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:34:32.975666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.975646    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:32.975743 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.975667    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75aa37b-8278-4dcc-8dbd-0911807a3791-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:32.977359 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.977338    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-provider-credentials" pod "c75aa37b-8278-4dcc-8dbd-0911807a3791" (UID: "c75aa37b-8278-4dcc-8dbd-0911807a3791"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:34:32.977503 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:32.977478    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-kube-api-access-x5t5r" pod "c75aa37b-8278-4dcc-8dbd-0911807a3791" (UID: "c75aa37b-8278-4dcc-8dbd-0911807a3791"). InnerVolumeSpecName "kube-api-access-x5t5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:34:33.076343 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.076307    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:33.076343 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.076337    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5t5r\" (UniqueName: \"kubernetes.io/projected/c75aa37b-8278-4dcc-8dbd-0911807a3791-kube-api-access-x5t5r\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:33.503446 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.503348    5510 generic.go:356] "Generic (PLEG): container finished" podID="c75aa37b-8278-4dcc-8dbd-0911807a3791" containerID="31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47" exitCode=0
Nov 01 06:34:33.503446 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.503398    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"
Nov 01 06:34:33.503668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.503444    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt" event={"ID":"c75aa37b-8278-4dcc-8dbd-0911807a3791","Type":"ContainerDied","Data":"31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47"}
Nov 01 06:34:33.503668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.503505    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt" event={"ID":"c75aa37b-8278-4dcc-8dbd-0911807a3791","Type":"ContainerDied","Data":"4ce38d0fc637d70a1f18c34274418cce706994acfce6ee64594736d6800cb87b"}
Nov 01 06:34:33.503668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.503538    5510 scope.go:122] "RemoveContainer" containerID="31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47"
Nov 01 06:34:33.513621 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.513586    5510 scope.go:122] "RemoveContainer" containerID="31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47"
Nov 01 06:34:33.513948 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:34:33.513928    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47\": container with ID starting with 31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47 not found: ID does not exist" containerID="31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47"
Nov 01 06:34:33.514019 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.513957    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47"} err="failed to get container status \"31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47\": rpc error: code = NotFound desc = could not find container \"31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47\": container with ID starting with 31eeb3cda723cd5d57857326ca823132a9247c7c0698c57391bf2cf314c34e47 not found: ID does not exist"
Nov 01 06:34:33.666561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.666527    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"]
Nov 01 06:34:33.715958 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:33.715924    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f658c5c64-p6rwt"]
Nov 01 06:34:35.236477 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:35.236448    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75aa37b-8278-4dcc-8dbd-0911807a3791" path="/var/lib/kubelet/pods/c75aa37b-8278-4dcc-8dbd-0911807a3791/volumes"
Nov 01 06:34:41.812353 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:41.812320    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:34:47.546078 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:34:47.546040    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:34:57.676655 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.676617    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"]
Nov 01 06:34:57.677324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.677304    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy"
Nov 01 06:34:57.677324 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.677324    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy"
Nov 01 06:34:57.677447 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.677346    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c75aa37b-8278-4dcc-8dbd-0911807a3791" containerName="payload-processing"
Nov 01 06:34:57.677447 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.677351    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75aa37b-8278-4dcc-8dbd-0911807a3791" containerName="payload-processing"
Nov 01 06:34:57.677447 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.677440    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75aa37b-8278-4dcc-8dbd-0911807a3791" containerName="payload-processing"
Nov 01 06:34:57.677619 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.677452    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3c5211-1885-4fbd-a6f9-da8f8ac2c7e5" containerName="istio-proxy"
Nov 01 06:34:57.682436 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.682416    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.719127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.719097    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-plugins-config-volume\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.719254 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.719142    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-provider-credentials\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.719300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.719287    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-routing-overlay-volume\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.719347 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.719336    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rnd\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-kube-api-access-w5rnd\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.734873 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.734849    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3bf320dd797d\""
Nov 01 06:34:57.734995 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.734882    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 06:34:57.784451 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.784408    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bc6f6b99b688\""
Nov 01 06:34:57.784451 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.784408    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 06:34:57.784451 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.784414    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1b1e343b358e\""
Nov 01 06:34:57.784451 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.784420    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 06:34:57.784756 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.784551    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qvhf2\""
Nov 01 06:34:57.784756 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.784576    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 06:34:57.787191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.787173    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"]
Nov 01 06:34:57.820692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.820668    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-plugins-config-volume\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.820865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.820721    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-provider-credentials\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.820865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.820803    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-routing-overlay-volume\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.820999 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.820870    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rnd\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-kube-api-access-w5rnd\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.821457 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.821431    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-plugins-config-volume\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.821745 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.821486    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-routing-overlay-volume\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.823536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.823508    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-provider-credentials\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.881333 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.881300    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rnd\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-kube-api-access-w5rnd\") pod \"payload-processing-external-model-78dccdd7cc-pgvt7\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") " pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:57.997184 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:57.997155    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:58.376739 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:58.376715    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"]
Nov 01 06:34:58.378628 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:34:58.378596    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b33733_8b25_497b_9f6c_dadca490f6f9.slice/crio-040271de88f16feb603f64a790e182682fc1b8b9f9b0eaff312ca6d0e5129acb WatchSource:0}: Error finding container 040271de88f16feb603f64a790e182682fc1b8b9f9b0eaff312ca6d0e5129acb: Status 404 returned error can't find the container with id 040271de88f16feb603f64a790e182682fc1b8b9f9b0eaff312ca6d0e5129acb
Nov 01 06:34:58.789790 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:58.789756    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7" event={"ID":"35b33733-8b25-497b-9f6c-dadca490f6f9","Type":"ContainerStarted","Data":"7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa"}
Nov 01 06:34:58.790329 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:58.789812    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7" event={"ID":"35b33733-8b25-497b-9f6c-dadca490f6f9","Type":"ContainerStarted","Data":"040271de88f16feb603f64a790e182682fc1b8b9f9b0eaff312ca6d0e5129acb"}
Nov 01 06:34:58.790329 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:58.789991    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:34:58.905573 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:34:58.905510    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7" podStartSLOduration=1.9054938780000001 podStartE2EDuration="1.905493878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:58.904961634 +0000 UTC m=+2935.782935046" watchObservedRunningTime="2026-11-01 06:34:58.905493878 +0000 UTC m=+2935.783467287"
Nov 01 06:35:09.801958 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:09.801928    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:35:17.544589 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:35:17.544553    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:35:23.210365 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:23.210319    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:35:34.396848 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.396799    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"]
Nov 01 06:35:34.397418 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.397033    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf" podUID="44b495b2-6210-4d04-845d-e942c97c4a3f" containerName="maas-api" containerID="cri-o://bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23" gracePeriod=30
Nov 01 06:35:34.718878 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.718847    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:35:34.794854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.792140    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-metrics-certs\") pod \"44b495b2-6210-4d04-845d-e942c97c4a3f\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") "
Nov 01 06:35:34.794854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.792194    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b495b2-6210-4d04-845d-e942c97c4a3f-kube-api-access-28s9w\" (UniqueName: \"kubernetes.io/projected/44b495b2-6210-4d04-845d-e942c97c4a3f-kube-api-access-28s9w\") pod \"44b495b2-6210-4d04-845d-e942c97c4a3f\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") "
Nov 01 06:35:34.794854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.792332    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls\") pod \"44b495b2-6210-4d04-845d-e942c97c4a3f\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") "
Nov 01 06:35:34.794854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.792422    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b495b2-6210-4d04-845d-e942c97c4a3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b495b2-6210-4d04-845d-e942c97c4a3f-openshift-service-ca\") pod \"44b495b2-6210-4d04-845d-e942c97c4a3f\" (UID: \"44b495b2-6210-4d04-845d-e942c97c4a3f\") "
Nov 01 06:35:34.827664 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.827540    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b495b2-6210-4d04-845d-e942c97c4a3f-kube-api-access-28s9w" pod "44b495b2-6210-4d04-845d-e942c97c4a3f" (UID: "44b495b2-6210-4d04-845d-e942c97c4a3f"). InnerVolumeSpecName "kube-api-access-28s9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:34.827664 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.827622    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-metrics-certs" pod "44b495b2-6210-4d04-845d-e942c97c4a3f" (UID: "44b495b2-6210-4d04-845d-e942c97c4a3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:35:34.827985 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.827948    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b495b2-6210-4d04-845d-e942c97c4a3f-openshift-service-ca" pod "44b495b2-6210-4d04-845d-e942c97c4a3f" (UID: "44b495b2-6210-4d04-845d-e942c97c4a3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:35:34.858132 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.856279    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls" pod "44b495b2-6210-4d04-845d-e942c97c4a3f" (UID: "44b495b2-6210-4d04-845d-e942c97c4a3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:35:34.893955 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.893913    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:34.893955 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.893953    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b495b2-6210-4d04-845d-e942c97c4a3f-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:34.894174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.893971    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b495b2-6210-4d04-845d-e942c97c4a3f-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:34.894174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:34.893984    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28s9w\" (UniqueName: \"kubernetes.io/projected/44b495b2-6210-4d04-845d-e942c97c4a3f-kube-api-access-28s9w\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:35.006606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.006572    5510 generic.go:356] "Generic (PLEG): container finished" podID="44b495b2-6210-4d04-845d-e942c97c4a3f" containerID="bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23" exitCode=0
Nov 01 06:35:35.006803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.006641    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"
Nov 01 06:35:35.006803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.006650    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf" event={"ID":"44b495b2-6210-4d04-845d-e942c97c4a3f","Type":"ContainerDied","Data":"bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23"}
Nov 01 06:35:35.006803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.006698    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf" event={"ID":"44b495b2-6210-4d04-845d-e942c97c4a3f","Type":"ContainerDied","Data":"a4aecf4e5afd095cf6bc73f876651f6a33f982f65ed0c4b72b101819cfeb23d6"}
Nov 01 06:35:35.006803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.006722    5510 scope.go:122] "RemoveContainer" containerID="bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23"
Nov 01 06:35:35.017170 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.017151    5510 scope.go:122] "RemoveContainer" containerID="bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23"
Nov 01 06:35:35.017421 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:35:35.017401    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23\": container with ID starting with bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23 not found: ID does not exist" containerID="bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23"
Nov 01 06:35:35.017461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.017431    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23"} err="failed to get container status \"bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23\": rpc error: code = NotFound desc = could not find container \"bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23\": container with ID starting with bc7f85589e7cca44f4eb28a658d5e5522a692bc7c955e57a7cd62ca3f218da23 not found: ID does not exist"
Nov 01 06:35:35.187285 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.187249    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"]
Nov 01 06:35:35.191117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.191095    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-tptqf"]
Nov 01 06:35:35.237398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:35.237367    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b495b2-6210-4d04-845d-e942c97c4a3f" path="/var/lib/kubelet/pods/44b495b2-6210-4d04-845d-e942c97c4a3f/volumes"
Nov 01 06:35:47.546151 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:35:47.546070    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:35:56.948416 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:56.948378    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"]
Nov 01 06:35:56.949051 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:56.949037    5510 cpu_manager.go:414] "containerMap: removing container" podUID="44b495b2-6210-4d04-845d-e942c97c4a3f" containerName="maas-api"
Nov 01 06:35:56.949109 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:56.949053    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b495b2-6210-4d04-845d-e942c97c4a3f" containerName="maas-api"
Nov 01 06:35:56.949164 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:56.949155    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b495b2-6210-4d04-845d-e942c97c4a3f" containerName="maas-api"
Nov 01 06:35:56.956489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:56.956472    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:56.962053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:56.962030    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2kl\" (UniqueName: \"kubernetes.io/projected/94a5e7d9-5206-4819-addd-af1ab9bb7731-kube-api-access-5p2kl\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn\" (UID: \"94a5e7d9-5206-4819-addd-af1ab9bb7731\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:56.962169 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:56.962069    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a5e7d9-5206-4819-addd-af1ab9bb7731-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn\" (UID: \"94a5e7d9-5206-4819-addd-af1ab9bb7731\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:57.006330 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.006305    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"]
Nov 01 06:35:57.008424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.008396    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c0662f25\""
Nov 01 06:35:57.062641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.062612    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2kl\" (UniqueName: \"kubernetes.io/projected/94a5e7d9-5206-4819-addd-af1ab9bb7731-kube-api-access-5p2kl\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn\" (UID: \"94a5e7d9-5206-4819-addd-af1ab9bb7731\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:57.062762 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.062662    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a5e7d9-5206-4819-addd-af1ab9bb7731-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn\" (UID: \"94a5e7d9-5206-4819-addd-af1ab9bb7731\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:57.063614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.063591    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a5e7d9-5206-4819-addd-af1ab9bb7731-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn\" (UID: \"94a5e7d9-5206-4819-addd-af1ab9bb7731\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:57.072468 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.072439    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:35:57.109358 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.109321    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"]
Nov 01 06:35:57.117258 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.116348    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:57.149870 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.142535    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2kl\" (UniqueName: \"kubernetes.io/projected/94a5e7d9-5206-4819-addd-af1ab9bb7731-kube-api-access-5p2kl\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn\" (UID: \"94a5e7d9-5206-4819-addd-af1ab9bb7731\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:57.165641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.163496    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"]
Nov 01 06:35:57.172207 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.172176    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c0662f25-dockercfg-7dlq2\""
Nov 01 06:35:57.265572 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.265542    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ffbd76-09ea-4a1b-a61c-cba78d911432-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-jp48g\" (UID: \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:57.265773 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.265684    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr9d4\" (UniqueName: \"kubernetes.io/projected/e0ffbd76-09ea-4a1b-a61c-cba78d911432-kube-api-access-pr9d4\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-jp48g\" (UID: \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:57.272321 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.272298    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:57.398928 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.398630    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ffbd76-09ea-4a1b-a61c-cba78d911432-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-jp48g\" (UID: \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:57.398928 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.391889    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ffbd76-09ea-4a1b-a61c-cba78d911432-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-jp48g\" (UID: \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:57.399186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.398996    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9d4\" (UniqueName: \"kubernetes.io/projected/e0ffbd76-09ea-4a1b-a61c-cba78d911432-kube-api-access-pr9d4\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-jp48g\" (UID: \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:57.464876 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.460839    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9d4\" (UniqueName: \"kubernetes.io/projected/e0ffbd76-09ea-4a1b-a61c-cba78d911432-kube-api-access-pr9d4\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-jp48g\" (UID: \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:57.745855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.745111    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"]
Nov 01 06:35:57.750847 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:57.747994    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:58.159326 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:58.159189    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn" event={"ID":"94a5e7d9-5206-4819-addd-af1ab9bb7731","Type":"ContainerStarted","Data":"5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a"}
Nov 01 06:35:58.159326 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:58.159293    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn" event={"ID":"94a5e7d9-5206-4819-addd-af1ab9bb7731","Type":"ContainerStarted","Data":"a67ec931a9f3ee754947d5cc74ea3a9af0fe3b6d60c6299032670599b9d0873f"}
Nov 01 06:35:58.159759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:58.159347    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:35:58.161202 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:58.161182    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"]
Nov 01 06:35:58.163685 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:35:58.163656    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ffbd76_09ea_4a1b_a61c_cba78d911432.slice/crio-085ec6f913bdad392d68822cc584b92b7cb1044498606e696f374c4c38236507 WatchSource:0}: Error finding container 085ec6f913bdad392d68822cc584b92b7cb1044498606e696f374c4c38236507: Status 404 returned error can't find the container with id 085ec6f913bdad392d68822cc584b92b7cb1044498606e696f374c4c38236507
Nov 01 06:35:58.275724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:58.275669    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn" podStartSLOduration=2.275653525 podStartE2EDuration="2.275653525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:58.273643251 +0000 UTC m=+2995.151616685" watchObservedRunningTime="2026-11-01 06:35:58.275653525 +0000 UTC m=+2995.153626938"
Nov 01 06:35:59.166038 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:59.166002    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g" event={"ID":"e0ffbd76-09ea-4a1b-a61c-cba78d911432","Type":"ContainerStarted","Data":"6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da"}
Nov 01 06:35:59.166451 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:59.166061    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g" event={"ID":"e0ffbd76-09ea-4a1b-a61c-cba78d911432","Type":"ContainerStarted","Data":"085ec6f913bdad392d68822cc584b92b7cb1044498606e696f374c4c38236507"}
Nov 01 06:35:59.166451 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:59.166090    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:35:59.283515 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:35:59.283458    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g" podStartSLOduration=2.283444187 podStartE2EDuration="2.283444187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:59.280642176 +0000 UTC m=+2996.158615607" watchObservedRunningTime="2026-11-01 06:35:59.283444187 +0000 UTC m=+2996.161417613"
Nov 01 06:36:09.171920 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:36:09.171889    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:36:10.175700 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:36:10.175668    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:36:14.275364 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:36:14.275327    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal" podUID="784c45e8dd43338b3f3bbbd833ce9232" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 01 06:36:17.547433 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:36:17.547399    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:36:39.477268 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:36:39.477229    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:36:47.546148 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:36:47.546112    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:37:17.545755 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:37:17.545670    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:37:17.885391 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:17.885307    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:37:32.545889 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.545859    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"]
Nov 01 06:37:32.552315 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.552293    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"]
Nov 01 06:37:32.552442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.552417    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.560203 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.560186    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.604379 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.604350    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c0662f25-openshift-default-dockercfg-lff96\""
Nov 01 06:37:32.664485 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.664440    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.664485 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.664505    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpw9p\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-kube-api-access-tpw9p\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.665149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.664623    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.665149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.664692    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.665149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.664843    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.665149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.664879    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.665149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.664917    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.665149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.664978    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.665149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665063    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.665149 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665128    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd706eba-9022-48b7-b56a-fded3f00555f-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.665522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665182    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.665522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665206    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd706eba-9022-48b7-b56a-fded3f00555f-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.665522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665289    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.665522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665318    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfwb5\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-kube-api-access-kfwb5\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.665522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665369    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.665522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665400    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.665522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665432    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.665522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.665473    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.667100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.667080    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"]
Nov 01 06:37:32.735321 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.735291    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"]
Nov 01 06:37:32.766615 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766574    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.766615 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766615    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.766882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766646    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpw9p\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-kube-api-access-tpw9p\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.766882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766677    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.766882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766722    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.766882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766761    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.766882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766789    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.766882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766847    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766888    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766915    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766956    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd706eba-9022-48b7-b56a-fded3f00555f-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.766997    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767019    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd706eba-9022-48b7-b56a-fded3f00555f-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767048    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767069    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767105    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfwb5\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-kube-api-access-kfwb5\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767157    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767185    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.767229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767222    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.767799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767267    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.767799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767481    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.767799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767504    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.768008 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.767947    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.768383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.768160    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd706eba-9022-48b7-b56a-fded3f00555f-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.768383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.768170    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.768383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.768309    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.768383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.768345    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.768676 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.768413    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.770030 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.770004    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.770339 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.770312    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.770526 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.770509    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.770884 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.770868    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd706eba-9022-48b7-b56a-fded3f00555f-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.827377 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.827319    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.827545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.827522    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpw9p\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-kube-api-access-tpw9p\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.828159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.828141    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-44d8p\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:32.828273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.828254    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfwb5\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-kube-api-access-kfwb5\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.870275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.870250    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:32.878198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:32.878178    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:33.300364 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.300339    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"]
Nov 01 06:37:33.302600 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:37:33.302571    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd706eba_9022_48b7_b56a_fded3f00555f.slice/crio-a3dfd75576b4316959b1c1b03a80e147eec4f1469128fe1e88ea0f1170ed877c WatchSource:0}: Error finding container a3dfd75576b4316959b1c1b03a80e147eec4f1469128fe1e88ea0f1170ed877c: Status 404 returned error can't find the container with id a3dfd75576b4316959b1c1b03a80e147eec4f1469128fe1e88ea0f1170ed877c
Nov 01 06:37:33.303837 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.303803    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"]
Nov 01 06:37:33.304423 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:37:33.304404    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfea8a2_c09c_4d1c_92ee_e9acea616981.slice/crio-769ff95f4d6290df1d0805a672c5f32820a62f3620d79a4d7d038b5015f298f8 WatchSource:0}: Error finding container 769ff95f4d6290df1d0805a672c5f32820a62f3620d79a4d7d038b5015f298f8: Status 404 returned error can't find the container with id 769ff95f4d6290df1d0805a672c5f32820a62f3620d79a4d7d038b5015f298f8
Nov 01 06:37:33.308995 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.308964    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:33.309080 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.309026    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:33.309080 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.309054    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:33.310286 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.310259    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:33.310338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.310319    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:33.310373 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.310352    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:33.729190 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.729146    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" event={"ID":"fd706eba-9022-48b7-b56a-fded3f00555f","Type":"ContainerStarted","Data":"47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762"}
Nov 01 06:37:33.729683 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.729210    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" event={"ID":"fd706eba-9022-48b7-b56a-fded3f00555f","Type":"ContainerStarted","Data":"a3dfd75576b4316959b1c1b03a80e147eec4f1469128fe1e88ea0f1170ed877c"}
Nov 01 06:37:33.731266 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.731237    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" event={"ID":"5dfea8a2-c09c-4d1c-92ee-e9acea616981","Type":"ContainerStarted","Data":"0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b"}
Nov 01 06:37:33.731388 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.731288    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" event={"ID":"5dfea8a2-c09c-4d1c-92ee-e9acea616981","Type":"ContainerStarted","Data":"769ff95f4d6290df1d0805a672c5f32820a62f3620d79a4d7d038b5015f298f8"}
Nov 01 06:37:33.856442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.853722    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podStartSLOduration=1.853701532 podStartE2EDuration="1.853701532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:33.848366892 +0000 UTC m=+3090.726340306" watchObservedRunningTime="2026-11-01 06:37:33.853701532 +0000 UTC m=+3090.731674944"
Nov 01 06:37:33.874605 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.874570    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:33.888996 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:33.879916    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:34.026634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:34.026494    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" podStartSLOduration=2.026471445 podStartE2EDuration="2.026471445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:33.978145575 +0000 UTC m=+3090.856118986" watchObservedRunningTime="2026-11-01 06:37:34.026471445 +0000 UTC m=+3090.904444867"
Nov 01 06:37:34.898855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:34.898657    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 06:37:34.898855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:34.898721    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 06:37:34.899428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:34.899101    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:37:34.899428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:34.899141    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 06:37:35.876854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:35.873157    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:37:35.876854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:35.873217    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 06:37:35.884855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:35.881175    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 06:37:35.884855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:35.881229    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 06:37:36.876851 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:36.873128    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:37:36.876851 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:36.873189    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 06:37:36.884852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:36.880922    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 06:37:36.884852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:36.880979    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 06:37:37.890854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:37.878271    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:37:37.890854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:37.878332    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 06:37:37.890854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:37.879001    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 06:37:37.890854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:37.879042    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 06:37:38.880853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:38.877161    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:37:38.880853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:38.877219    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 06:37:38.884852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:38.882120    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 06:37:38.884852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:38.882169    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 06:37:39.871142 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:39.871100    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:37:39.871548 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:39.871175    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 06:37:39.952853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:39.952073    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:39.953454 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:39.953430    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:39.954241 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:39.954217    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:37:40.875681 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:40.875646    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:40.876100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:40.875990    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:40.876657 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:37:40.876640    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:37:47.545995 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:37:47.545961    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:38:02.513928 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.513891    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:38:02.767682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.767605    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"]
Nov 01 06:38:02.773029 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.772989    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.863854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.861081    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"]
Nov 01 06:38:02.933082 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.930678    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.940339 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.940308    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.940639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.940620    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.940760 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.940745    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.940945 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.940928    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.941148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.941134    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.941289 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.941273    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.941419 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.941400    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwfw6\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-kube-api-access-mwfw6\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:02.941535 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:02.941522    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.042983 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.042896    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.042983 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.042978    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.043178 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.043019    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.043178 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.043043    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwfw6\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-kube-api-access-mwfw6\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.043178 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.043065    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.043178 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.043110    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.043178 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.043128    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.043342 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.043188    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.043342 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.043206    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.043990 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.043964    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.044289 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.044242    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.044507 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.044490    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.047171 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.047125    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.047843 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.047802    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.048085 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.048058    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.048464 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.048446    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.104658 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.104630    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.104820 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.104705    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwfw6\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-kube-api-access-mwfw6\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.388027 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.387957    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:03.857709 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.857678    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"]
Nov 01 06:38:03.858321 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:38:03.858295    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3eee81_41c9_4724_85f0_5c5eae595b3c.slice/crio-0a049da79dfb039f77edddb268889d18bb1d8b9d3fbe63c21afa0197ac7a2444 WatchSource:0}: Error finding container 0a049da79dfb039f77edddb268889d18bb1d8b9d3fbe63c21afa0197ac7a2444: Status 404 returned error can't find the container with id 0a049da79dfb039f77edddb268889d18bb1d8b9d3fbe63c21afa0197ac7a2444
Nov 01 06:38:03.861021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.860985    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:38:03.861104 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.861056    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:38:03.861104 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:03.861087    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:38:04.045528 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:04.045496    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" event={"ID":"bd3eee81-41c9-4724-85f0-5c5eae595b3c","Type":"ContainerStarted","Data":"f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb"}
Nov 01 06:38:04.045710 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:04.045560    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" event={"ID":"bd3eee81-41c9-4724-85f0-5c5eae595b3c","Type":"ContainerStarted","Data":"0a049da79dfb039f77edddb268889d18bb1d8b9d3fbe63c21afa0197ac7a2444"}
Nov 01 06:38:04.166184 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:04.166073    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" podStartSLOduration=2.1660576049999998 podStartE2EDuration="2.166057605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:04.164904687 +0000 UTC m=+3121.042878097" watchObservedRunningTime="2026-11-01 06:38:04.166057605 +0000 UTC m=+3121.044031015"
Nov 01 06:38:04.397888 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:04.392994    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:04.397888 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:04.393125    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 06:38:04.397888 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:04.393185    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 06:38:05.391754 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:05.391721    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 06:38:05.392376 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:05.392347    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 06:38:06.393855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:06.389223    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 06:38:06.393855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:06.389286    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 06:38:07.397882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:07.397291    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 06:38:07.397882 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:07.397355    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 06:38:08.390853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:08.390216    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 06:38:08.390853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:08.390280    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 06:38:09.389853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:09.389192    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 06:38:09.389853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:09.389267    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 06:38:10.451885 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:10.451821    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:10.452474 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:10.452448    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:10.454162 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:10.454139    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:38:11.084853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.084380    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"]
Nov 01 06:38:11.110912 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.109346    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:11.147024 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.144679    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"]
Nov 01 06:38:11.162720 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.162660    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c673e5-dockercfg-cxdkj\""
Nov 01 06:38:11.163037 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.162991    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c673e5\""
Nov 01 06:38:11.240555 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.240514    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv\" (UID: \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\") " pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:11.240739 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.240561    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzf7\" (UniqueName: \"kubernetes.io/projected/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-kube-api-access-7kzf7\") pod \"payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv\" (UID: \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\") " pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:11.342326 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.342234    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv\" (UID: \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\") " pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:11.342571 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.342546    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzf7\" (UniqueName: \"kubernetes.io/projected/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-kube-api-access-7kzf7\") pod \"payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv\" (UID: \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\") " pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:11.342929 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.342909    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv\" (UID: \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\") " pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:11.403660 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.403630    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzf7\" (UniqueName: \"kubernetes.io/projected/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-kube-api-access-7kzf7\") pod \"payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv\" (UID: \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\") " pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:11.443215 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.443178    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:11.823182 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:11.823154    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"]
Nov 01 06:38:11.824142 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:38:11.824111    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fef46b3_7049_4e40_ab18_66b6dfeb0b50.slice/crio-17ba36a2e2c2a5e198be7e64b86cefede7bcbd8ff9d6284d418ed3e4886514a8 WatchSource:0}: Error finding container 17ba36a2e2c2a5e198be7e64b86cefede7bcbd8ff9d6284d418ed3e4886514a8: Status 404 returned error can't find the container with id 17ba36a2e2c2a5e198be7e64b86cefede7bcbd8ff9d6284d418ed3e4886514a8
Nov 01 06:38:12.140384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:12.140281    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv" event={"ID":"8fef46b3-7049-4e40-ab18-66b6dfeb0b50","Type":"ContainerStarted","Data":"a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1"}
Nov 01 06:38:12.140384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:12.140345    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv" event={"ID":"8fef46b3-7049-4e40-ab18-66b6dfeb0b50","Type":"ContainerStarted","Data":"17ba36a2e2c2a5e198be7e64b86cefede7bcbd8ff9d6284d418ed3e4886514a8"}
Nov 01 06:38:12.140616 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:12.140572    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:12.257924 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:12.257859    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv" podStartSLOduration=1.257841462 podStartE2EDuration="1.257841462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:12.255356401 +0000 UTC m=+3129.133329811" watchObservedRunningTime="2026-11-01 06:38:12.257841462 +0000 UTC m=+3129.135814878"
Nov 01 06:38:17.545124 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:38:17.545081    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:38:23.149541 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:23.149513    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:38:47.545432 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:38:47.545343    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:38:50.836719 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:50.836685    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"]
Nov 01 06:38:50.942426 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:50.942388    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"]
Nov 01 06:38:55.837883 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:55.837843    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g" podUID="e0ffbd76-09ea-4a1b-a61c-cba78d911432" containerName="payload-processing" containerID="cri-o://6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da" gracePeriod=25
Nov 01 06:38:55.942789 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:55.942749    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn" podUID="94a5e7d9-5206-4819-addd-af1ab9bb7731" containerName="payload-pre-processing" containerID="cri-o://5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a" gracePeriod=25
Nov 01 06:38:56.112080 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.112017    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:38:56.204575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.204548    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ffbd76-09ea-4a1b-a61c-cba78d911432-kube-api-access-pr9d4\" (UniqueName: \"kubernetes.io/projected/e0ffbd76-09ea-4a1b-a61c-cba78d911432-kube-api-access-pr9d4\") pod \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\" (UID: \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\") "
Nov 01 06:38:56.204746 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.204734    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ffbd76-09ea-4a1b-a61c-cba78d911432-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ffbd76-09ea-4a1b-a61c-cba78d911432-plugins-config-volume\") pod \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\" (UID: \"e0ffbd76-09ea-4a1b-a61c-cba78d911432\") "
Nov 01 06:38:56.205109 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.205074    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ffbd76-09ea-4a1b-a61c-cba78d911432-plugins-config-volume" pod "e0ffbd76-09ea-4a1b-a61c-cba78d911432" (UID: "e0ffbd76-09ea-4a1b-a61c-cba78d911432"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:38:56.206767 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.206744    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ffbd76-09ea-4a1b-a61c-cba78d911432-kube-api-access-pr9d4" pod "e0ffbd76-09ea-4a1b-a61c-cba78d911432" (UID: "e0ffbd76-09ea-4a1b-a61c-cba78d911432"). InnerVolumeSpecName "kube-api-access-pr9d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:38:56.215213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.215195    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:38:56.305892 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.305859    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a5e7d9-5206-4819-addd-af1ab9bb7731-kube-api-access-5p2kl\" (UniqueName: \"kubernetes.io/projected/94a5e7d9-5206-4819-addd-af1ab9bb7731-kube-api-access-5p2kl\") pod \"94a5e7d9-5206-4819-addd-af1ab9bb7731\" (UID: \"94a5e7d9-5206-4819-addd-af1ab9bb7731\") "
Nov 01 06:38:56.306052 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.305949    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a5e7d9-5206-4819-addd-af1ab9bb7731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a5e7d9-5206-4819-addd-af1ab9bb7731-plugins-config-volume\") pod \"94a5e7d9-5206-4819-addd-af1ab9bb7731\" (UID: \"94a5e7d9-5206-4819-addd-af1ab9bb7731\") "
Nov 01 06:38:56.306618 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.306281    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a5e7d9-5206-4819-addd-af1ab9bb7731-plugins-config-volume" pod "94a5e7d9-5206-4819-addd-af1ab9bb7731" (UID: "94a5e7d9-5206-4819-addd-af1ab9bb7731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:38:56.306618 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.306606    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a5e7d9-5206-4819-addd-af1ab9bb7731-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:38:56.306777 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.306624    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr9d4\" (UniqueName: \"kubernetes.io/projected/e0ffbd76-09ea-4a1b-a61c-cba78d911432-kube-api-access-pr9d4\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:38:56.306777 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.306641    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ffbd76-09ea-4a1b-a61c-cba78d911432-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:38:56.308257 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.308233    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a5e7d9-5206-4819-addd-af1ab9bb7731-kube-api-access-5p2kl" pod "94a5e7d9-5206-4819-addd-af1ab9bb7731" (UID: "94a5e7d9-5206-4819-addd-af1ab9bb7731"). InnerVolumeSpecName "kube-api-access-5p2kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:38:56.408397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.408322    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2kl\" (UniqueName: \"kubernetes.io/projected/94a5e7d9-5206-4819-addd-af1ab9bb7731-kube-api-access-5p2kl\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:38:56.438046 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.438014    5510 generic.go:356] "Generic (PLEG): container finished" podID="94a5e7d9-5206-4819-addd-af1ab9bb7731" containerID="5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a" exitCode=0
Nov 01 06:38:56.438236 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.438053    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"
Nov 01 06:38:56.438236 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.438099    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn" event={"ID":"94a5e7d9-5206-4819-addd-af1ab9bb7731","Type":"ContainerDied","Data":"5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a"}
Nov 01 06:38:56.438236 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.438163    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn" event={"ID":"94a5e7d9-5206-4819-addd-af1ab9bb7731","Type":"ContainerDied","Data":"a67ec931a9f3ee754947d5cc74ea3a9af0fe3b6d60c6299032670599b9d0873f"}
Nov 01 06:38:56.438236 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.438193    5510 scope.go:122] "RemoveContainer" containerID="5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a"
Nov 01 06:38:56.439618 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.439595    5510 generic.go:356] "Generic (PLEG): container finished" podID="e0ffbd76-09ea-4a1b-a61c-cba78d911432" containerID="6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da" exitCode=0
Nov 01 06:38:56.439721 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.439627    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g" event={"ID":"e0ffbd76-09ea-4a1b-a61c-cba78d911432","Type":"ContainerDied","Data":"6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da"}
Nov 01 06:38:56.439721 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.439645    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"
Nov 01 06:38:56.439721 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.439665    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g" event={"ID":"e0ffbd76-09ea-4a1b-a61c-cba78d911432","Type":"ContainerDied","Data":"085ec6f913bdad392d68822cc584b92b7cb1044498606e696f374c4c38236507"}
Nov 01 06:38:56.449604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.449576    5510 scope.go:122] "RemoveContainer" containerID="5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a"
Nov 01 06:38:56.449887 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:38:56.449868    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a\": container with ID starting with 5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a not found: ID does not exist" containerID="5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a"
Nov 01 06:38:56.449967 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.449895    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a"} err="failed to get container status \"5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a\": rpc error: code = NotFound desc = could not find container \"5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a\": container with ID starting with 5e738990c98ee9f9756efd50f67fbd516c85e46b76e381c94f68331e43f4d46a not found: ID does not exist"
Nov 01 06:38:56.449967 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.449911    5510 scope.go:122] "RemoveContainer" containerID="6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da"
Nov 01 06:38:56.459213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.459196    5510 scope.go:122] "RemoveContainer" containerID="6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da"
Nov 01 06:38:56.459501 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:38:56.459483    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da\": container with ID starting with 6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da not found: ID does not exist" containerID="6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da"
Nov 01 06:38:56.459602 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.459510    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da"} err="failed to get container status \"6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da\": rpc error: code = NotFound desc = could not find container \"6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da\": container with ID starting with 6a00553a19f40a9f5b4e4a66ceb85d201674bbe19cf155764ac3418eeccdb8da not found: ID does not exist"
Nov 01 06:38:56.601728 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.601685    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"]
Nov 01 06:38:56.606604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.606576    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-2vngn"]
Nov 01 06:38:56.715413 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.715386    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"]
Nov 01 06:38:56.719223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:56.719199    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-jp48g"]
Nov 01 06:38:57.236424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:57.236390    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a5e7d9-5206-4819-addd-af1ab9bb7731" path="/var/lib/kubelet/pods/94a5e7d9-5206-4819-addd-af1ab9bb7731/volumes"
Nov 01 06:38:57.236792 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:38:57.236772    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ffbd76-09ea-4a1b-a61c-cba78d911432" path="/var/lib/kubelet/pods/e0ffbd76-09ea-4a1b-a61c-cba78d911432/volumes"
Nov 01 06:39:03.440540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.440508    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"]
Nov 01 06:39:03.441127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.441112    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e0ffbd76-09ea-4a1b-a61c-cba78d911432" containerName="payload-processing"
Nov 01 06:39:03.441176 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.441129    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ffbd76-09ea-4a1b-a61c-cba78d911432" containerName="payload-processing"
Nov 01 06:39:03.441176 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.441142    5510 cpu_manager.go:414] "containerMap: removing container" podUID="94a5e7d9-5206-4819-addd-af1ab9bb7731" containerName="payload-pre-processing"
Nov 01 06:39:03.441176 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.441147    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a5e7d9-5206-4819-addd-af1ab9bb7731" containerName="payload-pre-processing"
Nov 01 06:39:03.441270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.441248    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ffbd76-09ea-4a1b-a61c-cba78d911432" containerName="payload-processing"
Nov 01 06:39:03.441270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.441257    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a5e7d9-5206-4819-addd-af1ab9bb7731" containerName="payload-pre-processing"
Nov 01 06:39:03.444569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.444552    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.491736 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.491709    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-plugins-config-volume\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.491918 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.491752    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-routing-overlay-volume\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.491918 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.491858    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqzk\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-kube-api-access-2pqzk\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.492021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.491925    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-provider-credentials\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.551068 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.551037    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"]
Nov 01 06:39:03.593170 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.593129    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-plugins-config-volume\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.593328 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.593187    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-routing-overlay-volume\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.593328 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.593243    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqzk\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-kube-api-access-2pqzk\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.593328 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.593298    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-provider-credentials\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.593889 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.593864    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-routing-overlay-volume\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.593990 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.593888    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-plugins-config-volume\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.595925 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.595904    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-provider-credentials\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.654230 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.654204    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqzk\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-kube-api-access-2pqzk\") pod \"payload-processing-external-model-5fbc98587d-6stlj\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:03.758635 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:03.758611    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:04.137005 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:04.136981    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"]
Nov 01 06:39:04.139012 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:39:04.138982    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6e1498_7733_4548_8b63_5f3329e2dc2b.slice/crio-a9383c1b30ab9741334fb507718d30e90482af3085ab8a83d8dab3a98c4b141e WatchSource:0}: Error finding container a9383c1b30ab9741334fb507718d30e90482af3085ab8a83d8dab3a98c4b141e: Status 404 returned error can't find the container with id a9383c1b30ab9741334fb507718d30e90482af3085ab8a83d8dab3a98c4b141e
Nov 01 06:39:04.491887 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:04.491846    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj" event={"ID":"0b6e1498-7733-4548-8b63-5f3329e2dc2b","Type":"ContainerStarted","Data":"c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722"}
Nov 01 06:39:04.492327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:04.491898    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj" event={"ID":"0b6e1498-7733-4548-8b63-5f3329e2dc2b","Type":"ContainerStarted","Data":"a9383c1b30ab9741334fb507718d30e90482af3085ab8a83d8dab3a98c4b141e"}
Nov 01 06:39:04.492327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:04.492025    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:04.609254 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:04.609198    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj" podStartSLOduration=1.609180312 podStartE2EDuration="1.609180312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:39:04.606495886 +0000 UTC m=+3181.484469318" watchObservedRunningTime="2026-11-01 06:39:04.609180312 +0000 UTC m=+3181.487153726"
Nov 01 06:39:09.180759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.180729    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:39:09.404855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.404808    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"]
Nov 01 06:39:09.410518 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.410496    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.462343 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.462314    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c673e5-serving-cert\""
Nov 01 06:39:09.509690 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.509663    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"]
Nov 01 06:39:09.564189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.564154    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpns5\" (UniqueName: \"kubernetes.io/projected/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-kube-api-access-hpns5\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.564384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.564211    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-maas-api-tls\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.564384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.564238    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-metrics-certs\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.564384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.564344    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-openshift-service-ca\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.668545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.665766    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpns5\" (UniqueName: \"kubernetes.io/projected/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-kube-api-access-hpns5\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.668545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.666189    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-maas-api-tls\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.668545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.666234    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-metrics-certs\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.668545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.666288    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-openshift-service-ca\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.668932 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.668698    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-openshift-service-ca\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.670607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.670583    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-metrics-certs\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.671631 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.671611    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-maas-api-tls\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:09.763916 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:09.760578    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpns5\" (UniqueName: \"kubernetes.io/projected/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-kube-api-access-hpns5\") pod \"maas-api-e2e-explicit-c673e5-79b745db96-s5lcp\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:10.027876 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:10.027198    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:10.502856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:10.502367    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"]
Nov 01 06:39:10.529987 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:10.525191    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:39:10.592873 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:10.591400    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp" event={"ID":"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e","Type":"ContainerStarted","Data":"07b2f9927a000973b575959ffe44989e4351669c138d8dcad2021da7b3ce337e"}
Nov 01 06:39:12.657056 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:12.657019    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp" event={"ID":"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e","Type":"ContainerStarted","Data":"0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f"}
Nov 01 06:39:12.657896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:12.657869    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:12.776547 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:12.776492    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp" podStartSLOduration=2.779703063 podStartE2EDuration="3.776477406s" totalImagesPullingTime="996.774343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:39:12.774074185 +0000 UTC m=+3189.652047595" watchObservedRunningTime="2026-11-01 06:39:12.776477406 +0000 UTC m=+3189.654450816"
Nov 01 06:39:14.275372 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:14.275334    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"]
Nov 01 06:39:14.324479 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:14.324446    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"]
Nov 01 06:39:14.327788 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:14.327758    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj" podUID="0b6e1498-7733-4548-8b63-5f3329e2dc2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:39:17.545009 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:39:17.544976    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:39:19.276317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.276282    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7" podUID="35b33733-8b25-497b-9f6c-dadca490f6f9" containerName="payload-processing" containerID="cri-o://7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa" gracePeriod=25
Nov 01 06:39:19.325566 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.325533    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj" podUID="0b6e1498-7733-4548-8b63-5f3329e2dc2b" containerName="payload-processing" containerID="cri-o://c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722" gracePeriod=25
Nov 01 06:39:19.632922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.617078    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:39:19.675363 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.675335    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:39:19.680102 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.680081    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:19.685279 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.685261    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-routing-overlay-volume\") pod \"35b33733-8b25-497b-9f6c-dadca490f6f9\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") "
Nov 01 06:39:19.685359 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.685304    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-provider-credentials\") pod \"35b33733-8b25-497b-9f6c-dadca490f6f9\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") "
Nov 01 06:39:19.685428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.685381    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-kube-api-access-w5rnd\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-kube-api-access-w5rnd\") pod \"35b33733-8b25-497b-9f6c-dadca490f6f9\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") "
Nov 01 06:39:19.685520 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.685507    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-plugins-config-volume\") pod \"35b33733-8b25-497b-9f6c-dadca490f6f9\" (UID: \"35b33733-8b25-497b-9f6c-dadca490f6f9\") "
Nov 01 06:39:19.685680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.685659    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-routing-overlay-volume" pod "35b33733-8b25-497b-9f6c-dadca490f6f9" (UID: "35b33733-8b25-497b-9f6c-dadca490f6f9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:39:19.685779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.685765    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:19.685959 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.685936    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-plugins-config-volume" pod "35b33733-8b25-497b-9f6c-dadca490f6f9" (UID: "35b33733-8b25-497b-9f6c-dadca490f6f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:39:19.687659 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.687637    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-provider-credentials" pod "35b33733-8b25-497b-9f6c-dadca490f6f9" (UID: "35b33733-8b25-497b-9f6c-dadca490f6f9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:39:19.687659 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.687650    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-kube-api-access-w5rnd" pod "35b33733-8b25-497b-9f6c-dadca490f6f9" (UID: "35b33733-8b25-497b-9f6c-dadca490f6f9"). InnerVolumeSpecName "kube-api-access-w5rnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:39:19.701340 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.701314    5510 generic.go:356] "Generic (PLEG): container finished" podID="0b6e1498-7733-4548-8b63-5f3329e2dc2b" containerID="c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722" exitCode=0
Nov 01 06:39:19.701459 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.701360    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"
Nov 01 06:39:19.701459 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.701399    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj" event={"ID":"0b6e1498-7733-4548-8b63-5f3329e2dc2b","Type":"ContainerDied","Data":"c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722"}
Nov 01 06:39:19.701555 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.701454    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj" event={"ID":"0b6e1498-7733-4548-8b63-5f3329e2dc2b","Type":"ContainerDied","Data":"a9383c1b30ab9741334fb507718d30e90482af3085ab8a83d8dab3a98c4b141e"}
Nov 01 06:39:19.701555 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.701489    5510 scope.go:122] "RemoveContainer" containerID="c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722"
Nov 01 06:39:19.702992 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.702962    5510 generic.go:356] "Generic (PLEG): container finished" podID="35b33733-8b25-497b-9f6c-dadca490f6f9" containerID="7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa" exitCode=0
Nov 01 06:39:19.703262 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.703003    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"
Nov 01 06:39:19.703262 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.703000    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7" event={"ID":"35b33733-8b25-497b-9f6c-dadca490f6f9","Type":"ContainerDied","Data":"7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa"}
Nov 01 06:39:19.703262 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.703122    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7" event={"ID":"35b33733-8b25-497b-9f6c-dadca490f6f9","Type":"ContainerDied","Data":"040271de88f16feb603f64a790e182682fc1b8b9f9b0eaff312ca6d0e5129acb"}
Nov 01 06:39:19.713306 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.713283    5510 scope.go:122] "RemoveContainer" containerID="c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722"
Nov 01 06:39:19.713603 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:39:19.713571    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722\": container with ID starting with c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722 not found: ID does not exist" containerID="c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722"
Nov 01 06:39:19.713686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.713610    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722"} err="failed to get container status \"c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722\": rpc error: code = NotFound desc = could not find container \"c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722\": container with ID starting with c4251cd2e0493e844893e08a391a3d904ede1fd5e4266457c15912c5a0553722 not found: ID does not exist"
Nov 01 06:39:19.713686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.713627    5510 scope.go:122] "RemoveContainer" containerID="7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa"
Nov 01 06:39:19.723217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.723197    5510 scope.go:122] "RemoveContainer" containerID="7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa"
Nov 01 06:39:19.723530 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:39:19.723509    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa\": container with ID starting with 7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa not found: ID does not exist" containerID="7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa"
Nov 01 06:39:19.723577 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.723540    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa"} err="failed to get container status \"7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa\": rpc error: code = NotFound desc = could not find container \"7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa\": container with ID starting with 7e230f903a0d4e93860e4c450ecd29989899bfdf4ecc50fdef1ed0095b0309fa not found: ID does not exist"
Nov 01 06:39:19.786561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.786488    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-plugins-config-volume\") pod \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") "
Nov 01 06:39:19.786561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.786530    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-routing-overlay-volume\") pod \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") "
Nov 01 06:39:19.786811 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.786588    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-kube-api-access-2pqzk\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-kube-api-access-2pqzk\") pod \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") "
Nov 01 06:39:19.786811 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.786639    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-provider-credentials\") pod \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\" (UID: \"0b6e1498-7733-4548-8b63-5f3329e2dc2b\") "
Nov 01 06:39:19.786966 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.786944    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-plugins-config-volume" pod "0b6e1498-7733-4548-8b63-5f3329e2dc2b" (UID: "0b6e1498-7733-4548-8b63-5f3329e2dc2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:39:19.787033 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.786951    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-routing-overlay-volume" pod "0b6e1498-7733-4548-8b63-5f3329e2dc2b" (UID: "0b6e1498-7733-4548-8b63-5f3329e2dc2b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:39:19.787284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.787261    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:19.787596 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.787288    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5rnd\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-kube-api-access-w5rnd\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:19.787596 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.787307    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b33733-8b25-497b-9f6c-dadca490f6f9-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:19.787596 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.787321    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b33733-8b25-497b-9f6c-dadca490f6f9-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:19.787596 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.787335    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6e1498-7733-4548-8b63-5f3329e2dc2b-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:19.789005 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.788982    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-provider-credentials" pod "0b6e1498-7733-4548-8b63-5f3329e2dc2b" (UID: "0b6e1498-7733-4548-8b63-5f3329e2dc2b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:39:19.789111 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.789092    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-kube-api-access-2pqzk" pod "0b6e1498-7733-4548-8b63-5f3329e2dc2b" (UID: "0b6e1498-7733-4548-8b63-5f3329e2dc2b"). InnerVolumeSpecName "kube-api-access-2pqzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:39:19.888270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.888238    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pqzk\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-kube-api-access-2pqzk\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:19.888270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:19.888265    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b6e1498-7733-4548-8b63-5f3329e2dc2b-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:20.080748 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:20.077116    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"]
Nov 01 06:39:20.088865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:20.088812    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78dccdd7cc-pgvt7"]
Nov 01 06:39:20.192017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:20.191986    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"]
Nov 01 06:39:20.196243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:20.196216    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-6stlj"]
Nov 01 06:39:21.236988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:21.236952    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6e1498-7733-4548-8b63-5f3329e2dc2b" path="/var/lib/kubelet/pods/0b6e1498-7733-4548-8b63-5f3329e2dc2b/volumes"
Nov 01 06:39:21.237500 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:21.237480    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b33733-8b25-497b-9f6c-dadca490f6f9" path="/var/lib/kubelet/pods/35b33733-8b25-497b-9f6c-dadca490f6f9/volumes"
Nov 01 06:39:47.544191 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:39:47.544146    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:39:55.427878 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:55.427773    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"]
Nov 01 06:39:55.428363 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:55.428057    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy" containerID="cri-o://47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762" gracePeriod=30
Nov 01 06:39:55.441980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:55.441952    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"]
Nov 01 06:39:55.442178 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:55.442151    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy" containerID="cri-o://0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b" gracePeriod=30
Nov 01 06:39:55.463559 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:55.463537    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"]
Nov 01 06:39:55.463721 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:55.463703    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy" containerID="cri-o://f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb" gracePeriod=30
Nov 01 06:39:55.878525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:55.878494    5510 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:39:55.878704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:39:55.878545    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:40:00.746977 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.746941    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:40:00.780384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.780363    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:40:00.794132 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.794108    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:40:00.848761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.848728    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-certs\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.848926 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.848788    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-istio-token\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-istio-token\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.848926 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.848858    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-envoy\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.849021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.848956    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd706eba-9022-48b7-b56a-fded3f00555f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd706eba-9022-48b7-b56a-fded3f00555f-istio-podinfo\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.849021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.848990    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-certs" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.849099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849033    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-kube-api-access-kfwb5\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-kube-api-access-kfwb5\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.849099 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849074    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd706eba-9022-48b7-b56a-fded3f00555f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd706eba-9022-48b7-b56a-fded3f00555f-istiod-ca-cert\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.849190 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849109    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-socket\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.849190 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849173    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-credential-socket\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.849291 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849259    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-data\") pod \"fd706eba-9022-48b7-b56a-fded3f00555f\" (UID: \"fd706eba-9022-48b7-b56a-fded3f00555f\") "
Nov 01 06:40:00.849570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849481    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-socket" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.849570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849513    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-credential-socket" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.849727 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849660    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-data" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.849969 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849846    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.849969 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849873    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.849969 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849888    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.849969 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849902    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.849969 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.849946    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd706eba-9022-48b7-b56a-fded3f00555f-istiod-ca-cert" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:00.851623 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.851605    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-envoy" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.851722 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.851706    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-istio-token" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.851776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.851762    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd706eba-9022-48b7-b56a-fded3f00555f-istio-podinfo" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:00.851776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.851762    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-kube-api-access-kfwb5" pod "fd706eba-9022-48b7-b56a-fded3f00555f" (UID: "fd706eba-9022-48b7-b56a-fded3f00555f"). InnerVolumeSpecName "kube-api-access-kfwb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.950854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.950756    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-credential-socket\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.950854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.950807    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-data\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.951053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.950878    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-envoy\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.951053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.950910    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-kube-api-access-mwfw6\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-kube-api-access-mwfw6\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.951053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.950932    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-kube-api-access-tpw9p\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-kube-api-access-tpw9p\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.951053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.950958    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-podinfo\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.951053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951020    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-certs\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.951053 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951041    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-credential-socket" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951069    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istiod-ca-cert\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951089    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-data" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951158    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-socket\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951201    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-podinfo\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951228    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-token\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-token\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951273    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-socket\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951308    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-token\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-token\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951332    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-certs" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.951386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951379    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-certs\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.952127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951432    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-credential-socket\") pod \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\" (UID: \"5dfea8a2-c09c-4d1c-92ee-e9acea616981\") "
Nov 01 06:40:00.952127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951440    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istiod-ca-cert" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:00.952127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951456    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-socket" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.952127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951519    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-envoy\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.952127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951750    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-data\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.952127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951778    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istiod-ca-cert\") pod \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\" (UID: \"bd3eee81-41c9-4724-85f0-5c5eae595b3c\") "
Nov 01 06:40:00.952127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.951856    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-credential-socket" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.952127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952066    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-socket" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.952557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952365    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istiod-ca-cert" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:00.952557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952460    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.952557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952482    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.952557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952497    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.952557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952510    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.952557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952525    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.952557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952539    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.952557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952552    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd706eba-9022-48b7-b56a-fded3f00555f-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.953000 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952565    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.953000 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952579    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.953000 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952593    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd706eba-9022-48b7-b56a-fded3f00555f-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.953000 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952606    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.953000 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952619    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfwb5\" (UniqueName: \"kubernetes.io/projected/fd706eba-9022-48b7-b56a-fded3f00555f-kube-api-access-kfwb5\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.953000 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952632    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd706eba-9022-48b7-b56a-fded3f00555f-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.953000 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.952931    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-data" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.953397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.953014    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-certs" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.953759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.953732    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-kube-api-access-mwfw6" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "kube-api-access-mwfw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.954050 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.954023    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-kube-api-access-tpw9p" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "kube-api-access-tpw9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.954141 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.954112    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-envoy" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.954257 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.954240    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-podinfo" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:00.954511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.954492    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-token" pod "5dfea8a2-c09c-4d1c-92ee-e9acea616981" (UID: "5dfea8a2-c09c-4d1c-92ee-e9acea616981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.955037 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.955017    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-token" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.955089 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.955045    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-envoy" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.955488 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.955471    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-podinfo" pod "bd3eee81-41c9-4724-85f0-5c5eae595b3c" (UID: "bd3eee81-41c9-4724-85f0-5c5eae595b3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:00.988908 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:00.988880    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:40:01.022596 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.022570    5510 generic.go:356] "Generic (PLEG): container finished" podID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerID="f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb" exitCode=0
Nov 01 06:40:01.022751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.022623    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"
Nov 01 06:40:01.022751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.022650    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" event={"ID":"bd3eee81-41c9-4724-85f0-5c5eae595b3c","Type":"ContainerDied","Data":"f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb"}
Nov 01 06:40:01.022751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.022704    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7" event={"ID":"bd3eee81-41c9-4724-85f0-5c5eae595b3c","Type":"ContainerDied","Data":"0a049da79dfb039f77edddb268889d18bb1d8b9d3fbe63c21afa0197ac7a2444"}
Nov 01 06:40:01.022751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.022739    5510 scope.go:122] "RemoveContainer" containerID="f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb"
Nov 01 06:40:01.024221 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.024193    5510 generic.go:356] "Generic (PLEG): container finished" podID="fd706eba-9022-48b7-b56a-fded3f00555f" containerID="47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762" exitCode=0
Nov 01 06:40:01.024544 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.024246    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"
Nov 01 06:40:01.024544 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.024274    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" event={"ID":"fd706eba-9022-48b7-b56a-fded3f00555f","Type":"ContainerDied","Data":"47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762"}
Nov 01 06:40:01.024544 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.024310    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2" event={"ID":"fd706eba-9022-48b7-b56a-fded3f00555f","Type":"ContainerDied","Data":"a3dfd75576b4316959b1c1b03a80e147eec4f1469128fe1e88ea0f1170ed877c"}
Nov 01 06:40:01.025850 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.025795    5510 generic.go:356] "Generic (PLEG): container finished" podID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerID="0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b" exitCode=0
Nov 01 06:40:01.025952 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.025852    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" event={"ID":"5dfea8a2-c09c-4d1c-92ee-e9acea616981","Type":"ContainerDied","Data":"0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b"}
Nov 01 06:40:01.025952 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.025890    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"
Nov 01 06:40:01.025952 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.025903    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p" event={"ID":"5dfea8a2-c09c-4d1c-92ee-e9acea616981","Type":"ContainerDied","Data":"769ff95f4d6290df1d0805a672c5f32820a62f3620d79a4d7d038b5015f298f8"}
Nov 01 06:40:01.034853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.034804    5510 scope.go:122] "RemoveContainer" containerID="f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb"
Nov 01 06:40:01.035181 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:40:01.035144    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb\": container with ID starting with f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb not found: ID does not exist" containerID="f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb"
Nov 01 06:40:01.035271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.035193    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb"} err="failed to get container status \"f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb\": rpc error: code = NotFound desc = could not find container \"f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb\": container with ID starting with f47106d1861ab8cac09d5017e5dbc25df8c3d06452d22497d2b102154e3425bb not found: ID does not exist"
Nov 01 06:40:01.035271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.035216    5510 scope.go:122] "RemoveContainer" containerID="47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762"
Nov 01 06:40:01.046167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.046146    5510 scope.go:122] "RemoveContainer" containerID="47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762"
Nov 01 06:40:01.046456 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:40:01.046436    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762\": container with ID starting with 47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762 not found: ID does not exist" containerID="47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762"
Nov 01 06:40:01.046538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.046465    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762"} err="failed to get container status \"47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762\": rpc error: code = NotFound desc = could not find container \"47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762\": container with ID starting with 47ac88e141fe924431214876d392899a792f6549c940f47fe1bfa500e9a57762 not found: ID does not exist"
Nov 01 06:40:01.046538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.046489    5510 scope.go:122] "RemoveContainer" containerID="0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b"
Nov 01 06:40:01.054120 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054099    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054121    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054131    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054141    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwfw6\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-kube-api-access-mwfw6\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054150    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpw9p\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-kube-api-access-tpw9p\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054158    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054170    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054178    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfea8a2-c09c-4d1c-92ee-e9acea616981-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054186    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3eee81-41c9-4724-85f0-5c5eae595b3c-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.054204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.054194    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfea8a2-c09c-4d1c-92ee-e9acea616981-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.055668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.055653    5510 scope.go:122] "RemoveContainer" containerID="0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b"
Nov 01 06:40:01.055951 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:40:01.055932    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b\": container with ID starting with 0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b not found: ID does not exist" containerID="0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b"
Nov 01 06:40:01.056018 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.055966    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b"} err="failed to get container status \"0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b\": rpc error: code = NotFound desc = could not find container \"0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b\": container with ID starting with 0460ed934b72855f8cec8fb31c63cd16e90e7e72c94b2678aeedd752bfdb535b not found: ID does not exist"
Nov 01 06:40:01.271153 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.271120    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"]
Nov 01 06:40:01.273338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.273320    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-44d8p"]
Nov 01 06:40:01.381576 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.381535    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"]
Nov 01 06:40:01.385628 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.385601    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-ln2x2"]
Nov 01 06:40:01.495310 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.495280    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"]
Nov 01 06:40:01.499608 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:01.499585    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-7nkc7"]
Nov 01 06:40:03.244966 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:03.244931    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" path="/var/lib/kubelet/pods/5dfea8a2-c09c-4d1c-92ee-e9acea616981/volumes"
Nov 01 06:40:03.245380 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:03.245366    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" path="/var/lib/kubelet/pods/bd3eee81-41c9-4724-85f0-5c5eae595b3c/volumes"
Nov 01 06:40:03.245785 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:03.245772    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" path="/var/lib/kubelet/pods/fd706eba-9022-48b7-b56a-fded3f00555f/volumes"
Nov 01 06:40:17.545868 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:40:17.545766    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:40:27.139786 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:40:27.139747    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:40:47.545974 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:40:47.545936    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:41:13.202374 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:13.202340    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:41:17.545032 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:41:17.544992    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:41:47.544606 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:41:47.544527    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:41:48.272032 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:48.271997    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"]
Nov 01 06:41:53.273005 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.272969    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv" podUID="8fef46b3-7049-4e40-ab18-66b6dfeb0b50" containerName="payload-processing" containerID="cri-o://a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1" gracePeriod=25
Nov 01 06:41:53.528082 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.528025    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:41:53.630242 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.630215    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-kube-api-access-7kzf7\" (UniqueName: \"kubernetes.io/projected/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-kube-api-access-7kzf7\") pod \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\" (UID: \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\") "
Nov 01 06:41:53.630398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.630384    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-plugins-config-volume\") pod \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\" (UID: \"8fef46b3-7049-4e40-ab18-66b6dfeb0b50\") "
Nov 01 06:41:53.630703 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.630683    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-plugins-config-volume" pod "8fef46b3-7049-4e40-ab18-66b6dfeb0b50" (UID: "8fef46b3-7049-4e40-ab18-66b6dfeb0b50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:41:53.632543 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.632511    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-kube-api-access-7kzf7" pod "8fef46b3-7049-4e40-ab18-66b6dfeb0b50" (UID: "8fef46b3-7049-4e40-ab18-66b6dfeb0b50"). InnerVolumeSpecName "kube-api-access-7kzf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:41:53.731847 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.731799    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kzf7\" (UniqueName: \"kubernetes.io/projected/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-kube-api-access-7kzf7\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:41:53.732019 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.731862    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fef46b3-7049-4e40-ab18-66b6dfeb0b50-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:41:53.775770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.775737    5510 generic.go:356] "Generic (PLEG): container finished" podID="8fef46b3-7049-4e40-ab18-66b6dfeb0b50" containerID="a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1" exitCode=0
Nov 01 06:41:53.775947 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.775783    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"
Nov 01 06:41:53.775947 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.775850    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv" event={"ID":"8fef46b3-7049-4e40-ab18-66b6dfeb0b50","Type":"ContainerDied","Data":"a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1"}
Nov 01 06:41:53.775947 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.775904    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv" event={"ID":"8fef46b3-7049-4e40-ab18-66b6dfeb0b50","Type":"ContainerDied","Data":"17ba36a2e2c2a5e198be7e64b86cefede7bcbd8ff9d6284d418ed3e4886514a8"}
Nov 01 06:41:53.776081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.775927    5510 scope.go:122] "RemoveContainer" containerID="a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1"
Nov 01 06:41:53.788579 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.788560    5510 scope.go:122] "RemoveContainer" containerID="a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1"
Nov 01 06:41:53.793203 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:41:53.793175    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1\": container with ID starting with a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1 not found: ID does not exist" containerID="a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1"
Nov 01 06:41:53.793313 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.793214    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1"} err="failed to get container status \"a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1\": rpc error: code = NotFound desc = could not find container \"a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1\": container with ID starting with a2722bb05fca1f6b839fe6718b0e10e75a6fe929a11a2afc1d0edb08c1c97fe1 not found: ID does not exist"
Nov 01 06:41:53.939006 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.938960    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"]
Nov 01 06:41:53.941356 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:53.941329    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c673e5-7df6b77d9f-vvrgv"]
Nov 01 06:41:55.236464 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:41:55.236430    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fef46b3-7049-4e40-ab18-66b6dfeb0b50" path="/var/lib/kubelet/pods/8fef46b3-7049-4e40-ab18-66b6dfeb0b50/volumes"
Nov 01 06:42:12.374875 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:12.374841    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:42:17.546262 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:42:17.546223    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:42:18.181750 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.181716    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"]
Nov 01 06:42:18.182014 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.181988    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp" podUID="4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" containerName="maas-api" containerID="cri-o://0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f" gracePeriod=30
Nov 01 06:42:18.467193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.467171    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:42:18.631818 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.631786    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-kube-api-access-hpns5\" (UniqueName: \"kubernetes.io/projected/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-kube-api-access-hpns5\") pod \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") "
Nov 01 06:42:18.632270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.631871    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-maas-api-tls\") pod \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") "
Nov 01 06:42:18.632270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.631990    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-metrics-certs\") pod \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") "
Nov 01 06:42:18.632270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.632022    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-openshift-service-ca\") pod \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\" (UID: \"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e\") "
Nov 01 06:42:18.632456 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.632376    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-openshift-service-ca" pod "4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" (UID: "4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:42:18.632556 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.632521    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:18.634383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.634357    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-maas-api-tls" pod "4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" (UID: "4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:42:18.634747 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.634726    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-metrics-certs" pod "4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" (UID: "4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:42:18.636862 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.635100    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-kube-api-access-hpns5" pod "4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" (UID: "4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e"). InnerVolumeSpecName "kube-api-access-hpns5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:18.741693 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.740454    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpns5\" (UniqueName: \"kubernetes.io/projected/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-kube-api-access-hpns5\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:18.741693 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.740497    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:18.741693 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.740511    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:18.941228 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.941194    5510 generic.go:356] "Generic (PLEG): container finished" podID="4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" containerID="0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f" exitCode=0
Nov 01 06:42:18.941423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.941246    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"
Nov 01 06:42:18.941423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.941280    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp" event={"ID":"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e","Type":"ContainerDied","Data":"0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f"}
Nov 01 06:42:18.941423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.941348    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp" event={"ID":"4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e","Type":"ContainerDied","Data":"07b2f9927a000973b575959ffe44989e4351669c138d8dcad2021da7b3ce337e"}
Nov 01 06:42:18.941423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.941380    5510 scope.go:122] "RemoveContainer" containerID="0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f"
Nov 01 06:42:18.951952 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.951934    5510 scope.go:122] "RemoveContainer" containerID="0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f"
Nov 01 06:42:18.952223 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:42:18.952205    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f\": container with ID starting with 0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f not found: ID does not exist" containerID="0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f"
Nov 01 06:42:18.952269 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:18.952234    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f"} err="failed to get container status \"0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f\": rpc error: code = NotFound desc = could not find container \"0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f\": container with ID starting with 0555b613f4586de236a14b10f06f69015d87bb1ca3e42e8ff5ac17b1233a488f not found: ID does not exist"
Nov 01 06:42:19.120953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:19.118226    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"]
Nov 01 06:42:19.172450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:19.172403    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c673e5-79b745db96-s5lcp"]
Nov 01 06:42:19.235957 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:19.235928    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" path="/var/lib/kubelet/pods/4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e/volumes"
Nov 01 06:42:47.544062 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:42:47.544026    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:42:56.379380 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:42:56.379337    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:43:17.578465 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:43:17.578388    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:43:36.809093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:43:36.809057    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:43:47.546548 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:43:47.546499    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:44:13.780278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780243    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"]
Nov 01 06:44:13.780800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780766    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy"
Nov 01 06:44:13.780800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780776    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy"
Nov 01 06:44:13.780800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780788    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy"
Nov 01 06:44:13.780800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780794    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780807    5510 cpu_manager.go:414] "containerMap: removing container" podUID="4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" containerName="maas-api"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780812    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" containerName="maas-api"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780842    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780847    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780853    5510 cpu_manager.go:414] "containerMap: removing container" podUID="35b33733-8b25-497b-9f6c-dadca490f6f9" containerName="payload-processing"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780859    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b33733-8b25-497b-9f6c-dadca490f6f9" containerName="payload-processing"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780870    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8fef46b3-7049-4e40-ab18-66b6dfeb0b50" containerName="payload-processing"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780878    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fef46b3-7049-4e40-ab18-66b6dfeb0b50" containerName="payload-processing"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780902    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0b6e1498-7733-4548-8b63-5f3329e2dc2b" containerName="payload-processing"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780906    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6e1498-7733-4548-8b63-5f3329e2dc2b" containerName="payload-processing"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.780991    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6e1498-7733-4548-8b63-5f3329e2dc2b" containerName="payload-processing"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.781000    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfea8a2-c09c-4d1c-92ee-e9acea616981" containerName="istio-proxy"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.781008    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd706eba-9022-48b7-b56a-fded3f00555f" containerName="istio-proxy"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.781015    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2d53c6-007f-4cc7-8271-36ac7f1d2c5e" containerName="maas-api"
Nov 01 06:44:13.781020 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.781024    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3eee81-41c9-4724-85f0-5c5eae595b3c" containerName="istio-proxy"
Nov 01 06:44:13.781533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.781031    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b33733-8b25-497b-9f6c-dadca490f6f9" containerName="payload-processing"
Nov 01 06:44:13.781533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.781036    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fef46b3-7049-4e40-ab18-66b6dfeb0b50" containerName="payload-processing"
Nov 01 06:44:13.784423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.784406    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:13.885866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.885817    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-089523cf8a24\""
Nov 01 06:44:13.885866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.885817    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"openshift-service-ca.crt\""
Nov 01 06:44:13.886117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.885873    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"e2e-cfl-fd3b7b-provider-cred\""
Nov 01 06:44:13.886117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.885874    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"kube-root-ca.crt\""
Nov 01 06:44:13.886117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.885895    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"routing-overlay\""
Nov 01 06:44:13.886117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.885852    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8f45a2\""
Nov 01 06:44:13.886117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.886078    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-e2e--cb1d2723-dockercfg-zfdw5\""
Nov 01 06:44:13.888397 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.888377    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"]
Nov 01 06:44:13.910716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.910680    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:13.910890 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.910747    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mk6\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-kube-api-access-77mk6\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:13.910979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.910917    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:13.911061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:13.911042    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.012445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.012396    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.012639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.012489    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.012639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.012554    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.012639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.012594    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mk6\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-kube-api-access-77mk6\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.013260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.013232    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.013381 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.013325    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.015130 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.015110    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.073359 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.073278    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mk6\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-kube-api-access-77mk6\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.098484 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.098449    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.474157 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.474131    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"]
Nov 01 06:44:14.475956 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:44:14.475930    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ad5c4f_86ca_496c_8fa1_9b051075cda9.slice/crio-6945ae84fe5e1ffedda4610ed40ba94c961524a7635a3db7a3b1a591939a8c51 WatchSource:0}: Error finding container 6945ae84fe5e1ffedda4610ed40ba94c961524a7635a3db7a3b1a591939a8c51: Status 404 returned error can't find the container with id 6945ae84fe5e1ffedda4610ed40ba94c961524a7635a3db7a3b1a591939a8c51
Nov 01 06:44:14.720073 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.720036    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj" event={"ID":"64ad5c4f-86ca-496c-8fa1-9b051075cda9","Type":"ContainerStarted","Data":"3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826"}
Nov 01 06:44:14.720285 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.720084    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj" event={"ID":"64ad5c4f-86ca-496c-8fa1-9b051075cda9","Type":"ContainerStarted","Data":"6945ae84fe5e1ffedda4610ed40ba94c961524a7635a3db7a3b1a591939a8c51"}
Nov 01 06:44:14.720285 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.720243    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:14.837256 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:14.837155    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj" podStartSLOduration=1.837137427 podStartE2EDuration="1.837137427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:14.835434245 +0000 UTC m=+3491.713407658" watchObservedRunningTime="2026-11-01 06:44:14.837137427 +0000 UTC m=+3491.715110838"
Nov 01 06:44:17.546062 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:44:17.546025    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:44:25.729334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:25.729294    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj" podUID="64ad5c4f-86ca-496c-8fa1-9b051075cda9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:44:28.016412 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:28.016373    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"]
Nov 01 06:44:28.017857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:28.017814    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj" podUID="64ad5c4f-86ca-496c-8fa1-9b051075cda9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:44:31.985939 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:31.985906    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:44:33.018024 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.017897    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj" podUID="64ad5c4f-86ca-496c-8fa1-9b051075cda9" containerName="payload-processing" containerID="cri-o://3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826" gracePeriod=25
Nov 01 06:44:33.289889 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.289867    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:33.362986 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.362959    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-plugins-config-volume\") pod \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") "
Nov 01 06:44:33.363104 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.363011    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-routing-overlay-volume\") pod \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") "
Nov 01 06:44:33.363104 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.363051    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-provider-credentials\") pod \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") "
Nov 01 06:44:33.363194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.363161    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-kube-api-access-77mk6\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-kube-api-access-77mk6\") pod \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\" (UID: \"64ad5c4f-86ca-496c-8fa1-9b051075cda9\") "
Nov 01 06:44:33.363390 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.363364    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-routing-overlay-volume" pod "64ad5c4f-86ca-496c-8fa1-9b051075cda9" (UID: "64ad5c4f-86ca-496c-8fa1-9b051075cda9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:33.363390 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.363376    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-plugins-config-volume" pod "64ad5c4f-86ca-496c-8fa1-9b051075cda9" (UID: "64ad5c4f-86ca-496c-8fa1-9b051075cda9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:33.363546 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.363528    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:33.363592 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.363551    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64ad5c4f-86ca-496c-8fa1-9b051075cda9-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:33.365276 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.365252    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-provider-credentials" pod "64ad5c4f-86ca-496c-8fa1-9b051075cda9" (UID: "64ad5c4f-86ca-496c-8fa1-9b051075cda9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:33.365380 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.365336    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-kube-api-access-77mk6" pod "64ad5c4f-86ca-496c-8fa1-9b051075cda9" (UID: "64ad5c4f-86ca-496c-8fa1-9b051075cda9"). InnerVolumeSpecName "kube-api-access-77mk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:33.465195 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.465167    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mk6\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-kube-api-access-77mk6\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:33.465195 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.465194    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64ad5c4f-86ca-496c-8fa1-9b051075cda9-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:33.836517 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.836473    5510 generic.go:356] "Generic (PLEG): container finished" podID="64ad5c4f-86ca-496c-8fa1-9b051075cda9" containerID="3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826" exitCode=0
Nov 01 06:44:33.836724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.836527    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"
Nov 01 06:44:33.836724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.836562    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj" event={"ID":"64ad5c4f-86ca-496c-8fa1-9b051075cda9","Type":"ContainerDied","Data":"3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826"}
Nov 01 06:44:33.836724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.836651    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj" event={"ID":"64ad5c4f-86ca-496c-8fa1-9b051075cda9","Type":"ContainerDied","Data":"6945ae84fe5e1ffedda4610ed40ba94c961524a7635a3db7a3b1a591939a8c51"}
Nov 01 06:44:33.836724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.836682    5510 scope.go:122] "RemoveContainer" containerID="3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826"
Nov 01 06:44:33.847030 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.847013    5510 scope.go:122] "RemoveContainer" containerID="3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826"
Nov 01 06:44:33.847300 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:44:33.847278    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826\": container with ID starting with 3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826 not found: ID does not exist" containerID="3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826"
Nov 01 06:44:33.847365 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.847313    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826"} err="failed to get container status \"3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826\": rpc error: code = NotFound desc = could not find container \"3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826\": container with ID starting with 3899ca3bcd729af48a28cad52957a24fe292b97f09807694390406a91a6a9826 not found: ID does not exist"
Nov 01 06:44:33.999547 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:33.999512    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"]
Nov 01 06:44:34.003724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:34.003696    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-69992w5gj"]
Nov 01 06:44:35.236479 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:44:35.236446    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ad5c4f-86ca-496c-8fa1-9b051075cda9" path="/var/lib/kubelet/pods/64ad5c4f-86ca-496c-8fa1-9b051075cda9/volumes"
Nov 01 06:44:47.549841 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:44:47.549746    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:45:00.164981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.164947    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"]
Nov 01 06:45:00.165583 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.165568    5510 cpu_manager.go:414] "containerMap: removing container" podUID="64ad5c4f-86ca-496c-8fa1-9b051075cda9" containerName="payload-processing"
Nov 01 06:45:00.165637 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.165584    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ad5c4f-86ca-496c-8fa1-9b051075cda9" containerName="payload-processing"
Nov 01 06:45:00.165734 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.165724    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ad5c4f-86ca-496c-8fa1-9b051075cda9" containerName="payload-processing"
Nov 01 06:45:00.169186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.169167    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:00.189852 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.189810    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"]
Nov 01 06:45:00.193557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.193542    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"]
Nov 01 06:45:00.193695 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.193676    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:00.197350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.197329    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:00.214351 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.214328    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"]
Nov 01 06:45:00.218769 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.218751    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"]
Nov 01 06:45:00.218897 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.218884    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:00.222609 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.222594    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:00.237007 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.236987    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:45:00.244565 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.244545    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm5dg\" (UniqueName: \"kubernetes.io/projected/cad581a7-fb28-4443-8900-1b3083749f18-kube-api-access-hm5dg\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95\" (UID: \"cad581a7-fb28-4443-8900-1b3083749f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:00.244757 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.244740    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad581a7-fb28-4443-8900-1b3083749f18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95\" (UID: \"cad581a7-fb28-4443-8900-1b3083749f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:00.268896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.268838    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"]
Nov 01 06:45:00.323863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.322798    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"]
Nov 01 06:45:00.325695 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.325670    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"]
Nov 01 06:45:00.346759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.346732    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a7d4fb-9891-4610-82a5-e6e79602ff3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx\" (UID: \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:00.346889 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.346770    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm5dg\" (UniqueName: \"kubernetes.io/projected/cad581a7-fb28-4443-8900-1b3083749f18-kube-api-access-hm5dg\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95\" (UID: \"cad581a7-fb28-4443-8900-1b3083749f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:00.346889 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.346792    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f92m\" (UniqueName: \"kubernetes.io/projected/0a81091f-96ea-49e6-899c-d64803b91306-kube-api-access-2f92m\") pod \"maas-api-key-cleanup-29891925-bcln5\" (UID: \"0a81091f-96ea-49e6-899c-d64803b91306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:00.346976 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.346957    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4gb\" (UniqueName: \"kubernetes.io/projected/a54b12b6-9821-486a-ab48-84bbc3855ff4-kube-api-access-xl4gb\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj\" (UID: \"a54b12b6-9821-486a-ab48-84bbc3855ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:00.347014 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.346993    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrcsq\" (UniqueName: \"kubernetes.io/projected/5dacc97f-943c-45b4-a8d7-c2805a393c01-kube-api-access-qrcsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb\" (UID: \"5dacc97f-943c-45b4-a8d7-c2805a393c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:00.347079 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.347058    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dacc97f-943c-45b4-a8d7-c2805a393c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb\" (UID: \"5dacc97f-943c-45b4-a8d7-c2805a393c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:00.347115 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.347104    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54b12b6-9821-486a-ab48-84bbc3855ff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj\" (UID: \"a54b12b6-9821-486a-ab48-84bbc3855ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:00.347204 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.347190    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81091f-96ea-49e6-899c-d64803b91306-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891925-bcln5\" (UID: \"0a81091f-96ea-49e6-899c-d64803b91306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:00.347239 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.347220    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad581a7-fb28-4443-8900-1b3083749f18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95\" (UID: \"cad581a7-fb28-4443-8900-1b3083749f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:00.347273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.347240    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrrhd\" (UniqueName: \"kubernetes.io/projected/29a7d4fb-9891-4610-82a5-e6e79602ff3f-kube-api-access-xrrhd\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx\" (UID: \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:00.347841 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.347799    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad581a7-fb28-4443-8900-1b3083749f18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95\" (UID: \"cad581a7-fb28-4443-8900-1b3083749f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:00.365692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.365648    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"]
Nov 01 06:45:00.389011 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.388974    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"]
Nov 01 06:45:00.409256 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.409232    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm5dg\" (UniqueName: \"kubernetes.io/projected/cad581a7-fb28-4443-8900-1b3083749f18-kube-api-access-hm5dg\") pod \"maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95\" (UID: \"cad581a7-fb28-4443-8900-1b3083749f18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:00.448728 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.448655    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dacc97f-943c-45b4-a8d7-c2805a393c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb\" (UID: \"5dacc97f-943c-45b4-a8d7-c2805a393c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:00.448728 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.448699    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54b12b6-9821-486a-ab48-84bbc3855ff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj\" (UID: \"a54b12b6-9821-486a-ab48-84bbc3855ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:00.449064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.448756    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81091f-96ea-49e6-899c-d64803b91306-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891925-bcln5\" (UID: \"0a81091f-96ea-49e6-899c-d64803b91306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:00.449064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.448780    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrrhd\" (UniqueName: \"kubernetes.io/projected/29a7d4fb-9891-4610-82a5-e6e79602ff3f-kube-api-access-xrrhd\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx\" (UID: \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:00.449064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.448913    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a7d4fb-9891-4610-82a5-e6e79602ff3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx\" (UID: \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:00.449064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.448950    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f92m\" (UniqueName: \"kubernetes.io/projected/0a81091f-96ea-49e6-899c-d64803b91306-kube-api-access-2f92m\") pod \"maas-api-key-cleanup-29891925-bcln5\" (UID: \"0a81091f-96ea-49e6-899c-d64803b91306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:00.449327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.449063    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4gb\" (UniqueName: \"kubernetes.io/projected/a54b12b6-9821-486a-ab48-84bbc3855ff4-kube-api-access-xl4gb\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj\" (UID: \"a54b12b6-9821-486a-ab48-84bbc3855ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:00.449327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.449102    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrcsq\" (UniqueName: \"kubernetes.io/projected/5dacc97f-943c-45b4-a8d7-c2805a393c01-kube-api-access-qrcsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb\" (UID: \"5dacc97f-943c-45b4-a8d7-c2805a393c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:00.449480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.449452    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dacc97f-943c-45b4-a8d7-c2805a393c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb\" (UID: \"5dacc97f-943c-45b4-a8d7-c2805a393c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:00.449480 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.449469    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54b12b6-9821-486a-ab48-84bbc3855ff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj\" (UID: \"a54b12b6-9821-486a-ab48-84bbc3855ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:00.449644 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.449540    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81091f-96ea-49e6-899c-d64803b91306-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891925-bcln5\" (UID: \"0a81091f-96ea-49e6-899c-d64803b91306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:00.449644 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.449634    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a7d4fb-9891-4610-82a5-e6e79602ff3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx\" (UID: \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:00.482459 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.482434    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:00.510207 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.510183    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrrhd\" (UniqueName: \"kubernetes.io/projected/29a7d4fb-9891-4610-82a5-e6e79602ff3f-kube-api-access-xrrhd\") pod \"maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx\" (UID: \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:00.510499 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.510476    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f92m\" (UniqueName: \"kubernetes.io/projected/0a81091f-96ea-49e6-899c-d64803b91306-kube-api-access-2f92m\") pod \"maas-api-key-cleanup-29891925-bcln5\" (UID: \"0a81091f-96ea-49e6-899c-d64803b91306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:00.510561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.510476    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrcsq\" (UniqueName: \"kubernetes.io/projected/5dacc97f-943c-45b4-a8d7-c2805a393c01-kube-api-access-qrcsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb\" (UID: \"5dacc97f-943c-45b4-a8d7-c2805a393c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:00.511079 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.511046    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4gb\" (UniqueName: \"kubernetes.io/projected/a54b12b6-9821-486a-ab48-84bbc3855ff4-kube-api-access-xl4gb\") pod \"maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj\" (UID: \"a54b12b6-9821-486a-ab48-84bbc3855ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:00.514959 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.514938    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:00.533386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.533356    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:00.542190 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.542083    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:00.809442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.809410    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:00.869097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.869059    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"]
Nov 01 06:45:00.870404 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:45:00.870375    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad581a7_fb28_4443_8900_1b3083749f18.slice/crio-b308bf010b5929a592c0e2cfbf3c9133ba61b82379dd5faf095bc6a81113c4f3 WatchSource:0}: Error finding container b308bf010b5929a592c0e2cfbf3c9133ba61b82379dd5faf095bc6a81113c4f3: Status 404 returned error can't find the container with id b308bf010b5929a592c0e2cfbf3c9133ba61b82379dd5faf095bc6a81113c4f3
Nov 01 06:45:00.930197 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.930175    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"]
Nov 01 06:45:00.931071 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:45:00.931044    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a7d4fb_9891_4610_82a5_e6e79602ff3f.slice/crio-709f5a8f6b05ab1c89d03e93921fbae91df56ffc88f13a5a8618b6738da999e1 WatchSource:0}: Error finding container 709f5a8f6b05ab1c89d03e93921fbae91df56ffc88f13a5a8618b6738da999e1: Status 404 returned error can't find the container with id 709f5a8f6b05ab1c89d03e93921fbae91df56ffc88f13a5a8618b6738da999e1
Nov 01 06:45:00.949430 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.949402    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"]
Nov 01 06:45:00.952213 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:45:00.952190    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54b12b6_9821_486a_ab48_84bbc3855ff4.slice/crio-21bf8706bc71418c678c4d7b9121ce5b7303b9596f147c2fb03ccc0ddbe0f3e7 WatchSource:0}: Error finding container 21bf8706bc71418c678c4d7b9121ce5b7303b9596f147c2fb03ccc0ddbe0f3e7: Status 404 returned error can't find the container with id 21bf8706bc71418c678c4d7b9121ce5b7303b9596f147c2fb03ccc0ddbe0f3e7
Nov 01 06:45:00.977600 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:00.977575    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"]
Nov 01 06:45:01.029421 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:01.029386    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" event={"ID":"29a7d4fb-9891-4610-82a5-e6e79602ff3f","Type":"ContainerStarted","Data":"709f5a8f6b05ab1c89d03e93921fbae91df56ffc88f13a5a8618b6738da999e1"}
Nov 01 06:45:01.030759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:01.030731    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" event={"ID":"a54b12b6-9821-486a-ab48-84bbc3855ff4","Type":"ContainerStarted","Data":"21bf8706bc71418c678c4d7b9121ce5b7303b9596f147c2fb03ccc0ddbe0f3e7"}
Nov 01 06:45:01.032058 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:01.032038    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" event={"ID":"cad581a7-fb28-4443-8900-1b3083749f18","Type":"ContainerStarted","Data":"b308bf010b5929a592c0e2cfbf3c9133ba61b82379dd5faf095bc6a81113c4f3"}
Nov 01 06:45:01.072816 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:45:01.072754    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a81091f_96ea_49e6_899c_d64803b91306.slice/crio-04b8ebf844ccd4af5bf9a7db199cc6383625fc7228e54511f1d8abd26e9fef77 WatchSource:0}: Error finding container 04b8ebf844ccd4af5bf9a7db199cc6383625fc7228e54511f1d8abd26e9fef77: Status 404 returned error can't find the container with id 04b8ebf844ccd4af5bf9a7db199cc6383625fc7228e54511f1d8abd26e9fef77
Nov 01 06:45:01.187980 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:01.187955    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"]
Nov 01 06:45:01.189975 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:45:01.189941    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dacc97f_943c_45b4_a8d7_c2805a393c01.slice/crio-97df27e59bf8a5bab0f17635f0b133f7e067464313d9877ad53557e9832a75fa WatchSource:0}: Error finding container 97df27e59bf8a5bab0f17635f0b133f7e067464313d9877ad53557e9832a75fa: Status 404 returned error can't find the container with id 97df27e59bf8a5bab0f17635f0b133f7e067464313d9877ad53557e9832a75fa
Nov 01 06:45:02.040115 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.040025    5510 generic.go:356] "Generic (PLEG): container finished" podID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerID="2af98fab62d6d5f68bf0641b3cbe0d771a78b3957a549f1815397635d64ae49c" exitCode=60
Nov 01 06:45:02.040296 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.040122    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" event={"ID":"29a7d4fb-9891-4610-82a5-e6e79602ff3f","Type":"ContainerDied","Data":"2af98fab62d6d5f68bf0641b3cbe0d771a78b3957a549f1815397635d64ae49c"}
Nov 01 06:45:02.040409 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.040391    5510 scope.go:122] "RemoveContainer" containerID="2af98fab62d6d5f68bf0641b3cbe0d771a78b3957a549f1815397635d64ae49c"
Nov 01 06:45:02.041842 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.041799    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" event={"ID":"a54b12b6-9821-486a-ab48-84bbc3855ff4","Type":"ContainerStarted","Data":"8d4d2f674ed8d9afa208d371e4e9f80adbcce6e234d839b0c028008ff3d4022f"}
Nov 01 06:45:02.043568 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.043491    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" event={"ID":"cad581a7-fb28-4443-8900-1b3083749f18","Type":"ContainerStarted","Data":"d9894cd16c40a7e90bee2c75596c4d627cdfdd8d3fac3e9c5ba693ce5c6ed476"}
Nov 01 06:45:02.045045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.045021    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" event={"ID":"0a81091f-96ea-49e6-899c-d64803b91306","Type":"ContainerStarted","Data":"337a7d09bc974f27cb6dade6c9aae04571390c4e71a6c44cce01e2ca46aa0d00"}
Nov 01 06:45:02.045150 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.045060    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" event={"ID":"0a81091f-96ea-49e6-899c-d64803b91306","Type":"ContainerStarted","Data":"04b8ebf844ccd4af5bf9a7db199cc6383625fc7228e54511f1d8abd26e9fef77"}
Nov 01 06:45:02.046617 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.046597    5510 generic.go:356] "Generic (PLEG): container finished" podID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerID="27c126a636271348ccd1f78fa8d6f2050e1d82ea1d8deabbf31127f06ecb7fdd" exitCode=60
Nov 01 06:45:02.046694 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.046626    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" event={"ID":"5dacc97f-943c-45b4-a8d7-c2805a393c01","Type":"ContainerDied","Data":"27c126a636271348ccd1f78fa8d6f2050e1d82ea1d8deabbf31127f06ecb7fdd"}
Nov 01 06:45:02.046694 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.046663    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" event={"ID":"5dacc97f-943c-45b4-a8d7-c2805a393c01","Type":"ContainerStarted","Data":"97df27e59bf8a5bab0f17635f0b133f7e067464313d9877ad53557e9832a75fa"}
Nov 01 06:45:02.046896 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.046882    5510 scope.go:122] "RemoveContainer" containerID="27c126a636271348ccd1f78fa8d6f2050e1d82ea1d8deabbf31127f06ecb7fdd"
Nov 01 06:45:02.383802 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.383703    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" podStartSLOduration=2.383691215 podStartE2EDuration="2.383691215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:02.382367529 +0000 UTC m=+3539.260340939" watchObservedRunningTime="2026-11-01 06:45:02.383691215 +0000 UTC m=+3539.261664625"
Nov 01 06:45:02.497873 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.497807    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" podStartSLOduration=2.49779543 podStartE2EDuration="2.49779543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:02.496685391 +0000 UTC m=+3539.374658801" watchObservedRunningTime="2026-11-01 06:45:02.49779543 +0000 UTC m=+3539.375768839"
Nov 01 06:45:02.613400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:02.613344    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" podStartSLOduration=2.61332974 podStartE2EDuration="2.61332974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:02.611212255 +0000 UTC m=+3539.489185665" watchObservedRunningTime="2026-11-01 06:45:02.61332974 +0000 UTC m=+3539.491303150"
Nov 01 06:45:03.065622 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.065590    5510 generic.go:356] "Generic (PLEG): container finished" podID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerID="4c4e18db71a921308528c499690c806bf449d3a6a73a60259488d84fd86aed10" exitCode=60
Nov 01 06:45:03.065791 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.065677    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" event={"ID":"29a7d4fb-9891-4610-82a5-e6e79602ff3f","Type":"ContainerDied","Data":"4c4e18db71a921308528c499690c806bf449d3a6a73a60259488d84fd86aed10"}
Nov 01 06:45:03.065791 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.065768    5510 scope.go:122] "RemoveContainer" containerID="2af98fab62d6d5f68bf0641b3cbe0d771a78b3957a549f1815397635d64ae49c"
Nov 01 06:45:03.065984 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.065966    5510 scope.go:122] "RemoveContainer" containerID="4c4e18db71a921308528c499690c806bf449d3a6a73a60259488d84fd86aed10"
Nov 01 06:45:03.066305 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:03.066284    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx_odh-ai-gateway-infra(29a7d4fb-9891-4610-82a5-e6e79602ff3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f"
Nov 01 06:45:03.067688 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.067665    5510 generic.go:356] "Generic (PLEG): container finished" podID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerID="8d4d2f674ed8d9afa208d371e4e9f80adbcce6e234d839b0c028008ff3d4022f" exitCode=60
Nov 01 06:45:03.067800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.067746    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" event={"ID":"a54b12b6-9821-486a-ab48-84bbc3855ff4","Type":"ContainerDied","Data":"8d4d2f674ed8d9afa208d371e4e9f80adbcce6e234d839b0c028008ff3d4022f"}
Nov 01 06:45:03.067971 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.067958    5510 scope.go:122] "RemoveContainer" containerID="8d4d2f674ed8d9afa208d371e4e9f80adbcce6e234d839b0c028008ff3d4022f"
Nov 01 06:45:03.069867 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.069463    5510 generic.go:356] "Generic (PLEG): container finished" podID="cad581a7-fb28-4443-8900-1b3083749f18" containerID="d9894cd16c40a7e90bee2c75596c4d627cdfdd8d3fac3e9c5ba693ce5c6ed476" exitCode=60
Nov 01 06:45:03.069867 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.069595    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" event={"ID":"cad581a7-fb28-4443-8900-1b3083749f18","Type":"ContainerDied","Data":"d9894cd16c40a7e90bee2c75596c4d627cdfdd8d3fac3e9c5ba693ce5c6ed476"}
Nov 01 06:45:03.069867 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.069712    5510 scope.go:122] "RemoveContainer" containerID="d9894cd16c40a7e90bee2c75596c4d627cdfdd8d3fac3e9c5ba693ce5c6ed476"
Nov 01 06:45:03.071396 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.071377    5510 generic.go:356] "Generic (PLEG): container finished" podID="0a81091f-96ea-49e6-899c-d64803b91306" containerID="337a7d09bc974f27cb6dade6c9aae04571390c4e71a6c44cce01e2ca46aa0d00" exitCode=60
Nov 01 06:45:03.071505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.071489    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" event={"ID":"0a81091f-96ea-49e6-899c-d64803b91306","Type":"ContainerDied","Data":"337a7d09bc974f27cb6dade6c9aae04571390c4e71a6c44cce01e2ca46aa0d00"}
Nov 01 06:45:03.072070 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.071821    5510 scope.go:122] "RemoveContainer" containerID="337a7d09bc974f27cb6dade6c9aae04571390c4e71a6c44cce01e2ca46aa0d00"
Nov 01 06:45:03.082738 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.082312    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" event={"ID":"5dacc97f-943c-45b4-a8d7-c2805a393c01","Type":"ContainerStarted","Data":"2d0866afb73b878e0332070bfd3fa7cdd4c1daa25a67ac4c5ab7767119643a5e"}
Nov 01 06:45:03.642504 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:03.642396    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" podStartSLOduration=3.6423834040000003 podStartE2EDuration="3.642383404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.640258765 +0000 UTC m=+3540.518232169" watchObservedRunningTime="2026-11-01 06:45:03.642383404 +0000 UTC m=+3540.520356814"
Nov 01 06:45:04.091061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.091027    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" event={"ID":"0a81091f-96ea-49e6-899c-d64803b91306","Type":"ContainerStarted","Data":"385e60f18a1bc23ed538478bbc5b918e7cda8ff39b39dd2c4de539c66aaf6cf0"}
Nov 01 06:45:04.092953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.092917    5510 generic.go:356] "Generic (PLEG): container finished" podID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerID="2d0866afb73b878e0332070bfd3fa7cdd4c1daa25a67ac4c5ab7767119643a5e" exitCode=60
Nov 01 06:45:04.093081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.092994    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" event={"ID":"5dacc97f-943c-45b4-a8d7-c2805a393c01","Type":"ContainerDied","Data":"2d0866afb73b878e0332070bfd3fa7cdd4c1daa25a67ac4c5ab7767119643a5e"}
Nov 01 06:45:04.093081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.093041    5510 scope.go:122] "RemoveContainer" containerID="27c126a636271348ccd1f78fa8d6f2050e1d82ea1d8deabbf31127f06ecb7fdd"
Nov 01 06:45:04.093233 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.093219    5510 scope.go:122] "RemoveContainer" containerID="2d0866afb73b878e0332070bfd3fa7cdd4c1daa25a67ac4c5ab7767119643a5e"
Nov 01 06:45:04.093501 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:04.093482    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb_odh-ai-gateway-infra(5dacc97f-943c-45b4-a8d7-c2805a393c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01"
Nov 01 06:45:04.095019 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.095002    5510 scope.go:122] "RemoveContainer" containerID="4c4e18db71a921308528c499690c806bf449d3a6a73a60259488d84fd86aed10"
Nov 01 06:45:04.095234 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:04.095217    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx_odh-ai-gateway-infra(29a7d4fb-9891-4610-82a5-e6e79602ff3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f"
Nov 01 06:45:04.096643 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.096622    5510 generic.go:356] "Generic (PLEG): container finished" podID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerID="6566216823b2b3a7ab0cfd1645b6b3f5427d7727bc2ed3fe5b65db0144c5d4c3" exitCode=60
Nov 01 06:45:04.096719 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.096700    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" event={"ID":"a54b12b6-9821-486a-ab48-84bbc3855ff4","Type":"ContainerDied","Data":"6566216823b2b3a7ab0cfd1645b6b3f5427d7727bc2ed3fe5b65db0144c5d4c3"}
Nov 01 06:45:04.096891 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.096879    5510 scope.go:122] "RemoveContainer" containerID="6566216823b2b3a7ab0cfd1645b6b3f5427d7727bc2ed3fe5b65db0144c5d4c3"
Nov 01 06:45:04.097067 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:04.097049    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj_odh-ai-gateway-infra(a54b12b6-9821-486a-ab48-84bbc3855ff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4"
Nov 01 06:45:04.098554 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.098533    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" event={"ID":"cad581a7-fb28-4443-8900-1b3083749f18","Type":"ContainerStarted","Data":"86d19f886b4fd5ab88885c71387be06af34bc77b233b98b6acf3f96b16839eca"}
Nov 01 06:45:04.105634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:04.105615    5510 scope.go:122] "RemoveContainer" containerID="8d4d2f674ed8d9afa208d371e4e9f80adbcce6e234d839b0c028008ff3d4022f"
Nov 01 06:45:05.106592 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.106563    5510 scope.go:122] "RemoveContainer" containerID="6566216823b2b3a7ab0cfd1645b6b3f5427d7727bc2ed3fe5b65db0144c5d4c3"
Nov 01 06:45:05.107063 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:05.106844    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj_odh-ai-gateway-infra(a54b12b6-9821-486a-ab48-84bbc3855ff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4"
Nov 01 06:45:05.108203 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.108177    5510 generic.go:356] "Generic (PLEG): container finished" podID="cad581a7-fb28-4443-8900-1b3083749f18" containerID="86d19f886b4fd5ab88885c71387be06af34bc77b233b98b6acf3f96b16839eca" exitCode=60
Nov 01 06:45:05.108289 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.108263    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" event={"ID":"cad581a7-fb28-4443-8900-1b3083749f18","Type":"ContainerDied","Data":"86d19f886b4fd5ab88885c71387be06af34bc77b233b98b6acf3f96b16839eca"}
Nov 01 06:45:05.108381 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.108368    5510 scope.go:122] "RemoveContainer" containerID="d9894cd16c40a7e90bee2c75596c4d627cdfdd8d3fac3e9c5ba693ce5c6ed476"
Nov 01 06:45:05.108465 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.108450    5510 scope.go:122] "RemoveContainer" containerID="86d19f886b4fd5ab88885c71387be06af34bc77b233b98b6acf3f96b16839eca"
Nov 01 06:45:05.108702 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:05.108682    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95_odh-ai-gateway-infra(cad581a7-fb28-4443-8900-1b3083749f18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" podUID="cad581a7-fb28-4443-8900-1b3083749f18"
Nov 01 06:45:05.110528 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.110508    5510 generic.go:356] "Generic (PLEG): container finished" podID="0a81091f-96ea-49e6-899c-d64803b91306" containerID="385e60f18a1bc23ed538478bbc5b918e7cda8ff39b39dd2c4de539c66aaf6cf0" exitCode=60
Nov 01 06:45:05.110764 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.110746    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" event={"ID":"0a81091f-96ea-49e6-899c-d64803b91306","Type":"ContainerDied","Data":"385e60f18a1bc23ed538478bbc5b918e7cda8ff39b39dd2c4de539c66aaf6cf0"}
Nov 01 06:45:05.111223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.111203    5510 scope.go:122] "RemoveContainer" containerID="385e60f18a1bc23ed538478bbc5b918e7cda8ff39b39dd2c4de539c66aaf6cf0"
Nov 01 06:45:05.111435 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:05.111415    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891925-bcln5_odh-ai-gateway-infra(0a81091f-96ea-49e6-899c-d64803b91306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" podUID="0a81091f-96ea-49e6-899c-d64803b91306"
Nov 01 06:45:05.115442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.115425    5510 scope.go:122] "RemoveContainer" containerID="2d0866afb73b878e0332070bfd3fa7cdd4c1daa25a67ac4c5ab7767119643a5e"
Nov 01 06:45:05.115613 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:05.115596    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb_odh-ai-gateway-infra(5dacc97f-943c-45b4-a8d7-c2805a393c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01"
Nov 01 06:45:05.123616 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:05.123593    5510 scope.go:122] "RemoveContainer" containerID="337a7d09bc974f27cb6dade6c9aae04571390c4e71a6c44cce01e2ca46aa0d00"
Nov 01 06:45:06.124110 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:06.124082    5510 scope.go:122] "RemoveContainer" containerID="86d19f886b4fd5ab88885c71387be06af34bc77b233b98b6acf3f96b16839eca"
Nov 01 06:45:06.124615 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:06.124382    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95_odh-ai-gateway-infra(cad581a7-fb28-4443-8900-1b3083749f18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" podUID="cad581a7-fb28-4443-8900-1b3083749f18"
Nov 01 06:45:06.125917 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:06.125899    5510 scope.go:122] "RemoveContainer" containerID="385e60f18a1bc23ed538478bbc5b918e7cda8ff39b39dd2c4de539c66aaf6cf0"
Nov 01 06:45:06.126124 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:06.126108    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891925-bcln5_odh-ai-gateway-infra(0a81091f-96ea-49e6-899c-d64803b91306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" podUID="0a81091f-96ea-49e6-899c-d64803b91306"
Nov 01 06:45:10.766974 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:10.766936    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:45:14.229562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:14.229528    5510 scope.go:122] "RemoveContainer" containerID="4c4e18db71a921308528c499690c806bf449d3a6a73a60259488d84fd86aed10"
Nov 01 06:45:15.182154 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:15.182119    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" event={"ID":"29a7d4fb-9891-4610-82a5-e6e79602ff3f","Type":"ContainerStarted","Data":"b613f57b19174ec6cbd29532f341eab989f9b959e55f8772fd1e8439fa80012f"}
Nov 01 06:45:15.182630 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:15.182531    5510 scope.go:122] "RemoveContainer" containerID="b613f57b19174ec6cbd29532f341eab989f9b959e55f8772fd1e8439fa80012f"
Nov 01 06:45:15.182862 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:15.182821    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx_odh-ai-gateway-infra(29a7d4fb-9891-4610-82a5-e6e79602ff3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f"
Nov 01 06:45:15.228792 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:15.228757    5510 scope.go:122] "RemoveContainer" containerID="6566216823b2b3a7ab0cfd1645b6b3f5427d7727bc2ed3fe5b65db0144c5d4c3"
Nov 01 06:45:15.229007 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:15.228846    5510 scope.go:122] "RemoveContainer" containerID="2d0866afb73b878e0332070bfd3fa7cdd4c1daa25a67ac4c5ab7767119643a5e"
Nov 01 06:45:16.189585 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.189541    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" event={"ID":"a54b12b6-9821-486a-ab48-84bbc3855ff4","Type":"ContainerStarted","Data":"7b7942636df375b5e960b36e2456adf4b062171a7bf1f985f5f98cccc569aa27"}
Nov 01 06:45:16.191460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.191439    5510 generic.go:356] "Generic (PLEG): container finished" podID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerID="cfcede506d317ff5d16de62dd5a012150a3835bcb4b94a16e5e2ba3109c5132c" exitCode=60
Nov 01 06:45:16.191557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.191517    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" event={"ID":"5dacc97f-943c-45b4-a8d7-c2805a393c01","Type":"ContainerDied","Data":"cfcede506d317ff5d16de62dd5a012150a3835bcb4b94a16e5e2ba3109c5132c"}
Nov 01 06:45:16.191603 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.191572    5510 scope.go:122] "RemoveContainer" containerID="2d0866afb73b878e0332070bfd3fa7cdd4c1daa25a67ac4c5ab7767119643a5e"
Nov 01 06:45:16.191924 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.191904    5510 scope.go:122] "RemoveContainer" containerID="cfcede506d317ff5d16de62dd5a012150a3835bcb4b94a16e5e2ba3109c5132c"
Nov 01 06:45:16.192191 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:16.192172    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb_odh-ai-gateway-infra(5dacc97f-943c-45b4-a8d7-c2805a393c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01"
Nov 01 06:45:16.193492 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.193475    5510 generic.go:356] "Generic (PLEG): container finished" podID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerID="b613f57b19174ec6cbd29532f341eab989f9b959e55f8772fd1e8439fa80012f" exitCode=60
Nov 01 06:45:16.193571 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.193505    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" event={"ID":"29a7d4fb-9891-4610-82a5-e6e79602ff3f","Type":"ContainerDied","Data":"b613f57b19174ec6cbd29532f341eab989f9b959e55f8772fd1e8439fa80012f"}
Nov 01 06:45:16.193775 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.193763    5510 scope.go:122] "RemoveContainer" containerID="b613f57b19174ec6cbd29532f341eab989f9b959e55f8772fd1e8439fa80012f"
Nov 01 06:45:16.193982 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:16.193968    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx_odh-ai-gateway-infra(29a7d4fb-9891-4610-82a5-e6e79602ff3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f"
Nov 01 06:45:16.220194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.220173    5510 scope.go:122] "RemoveContainer" containerID="4c4e18db71a921308528c499690c806bf449d3a6a73a60259488d84fd86aed10"
Nov 01 06:45:16.311102 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:16.311073    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"]
Nov 01 06:45:17.203205 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.203179    5510 generic.go:356] "Generic (PLEG): container finished" podID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerID="7b7942636df375b5e960b36e2456adf4b062171a7bf1f985f5f98cccc569aa27" exitCode=60
Nov 01 06:45:17.203607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.203253    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" event={"ID":"a54b12b6-9821-486a-ab48-84bbc3855ff4","Type":"ContainerDied","Data":"7b7942636df375b5e960b36e2456adf4b062171a7bf1f985f5f98cccc569aa27"}
Nov 01 06:45:17.203607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.203315    5510 scope.go:122] "RemoveContainer" containerID="6566216823b2b3a7ab0cfd1645b6b3f5427d7727bc2ed3fe5b65db0144c5d4c3"
Nov 01 06:45:17.203727 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.203673    5510 scope.go:122] "RemoveContainer" containerID="7b7942636df375b5e960b36e2456adf4b062171a7bf1f985f5f98cccc569aa27"
Nov 01 06:45:17.204040 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:17.204018    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj_odh-ai-gateway-infra(a54b12b6-9821-486a-ab48-84bbc3855ff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4"
Nov 01 06:45:17.245419 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.245390    5510 scope.go:122] "RemoveContainer" containerID="86d19f886b4fd5ab88885c71387be06af34bc77b233b98b6acf3f96b16839eca"
Nov 01 06:45:17.322887 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.322810    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"]
Nov 01 06:45:17.387978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.387946    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:17.478791 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.478756    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a7d4fb-9891-4610-82a5-e6e79602ff3f-kube-api-access-xrrhd\" (UniqueName: \"kubernetes.io/projected/29a7d4fb-9891-4610-82a5-e6e79602ff3f-kube-api-access-xrrhd\") pod \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\" (UID: \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\") "
Nov 01 06:45:17.479023 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.478878    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a7d4fb-9891-4610-82a5-e6e79602ff3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a7d4fb-9891-4610-82a5-e6e79602ff3f-maas-api-service-ca\") pod \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\" (UID: \"29a7d4fb-9891-4610-82a5-e6e79602ff3f\") "
Nov 01 06:45:17.479229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.479205    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a7d4fb-9891-4610-82a5-e6e79602ff3f-maas-api-service-ca" pod "29a7d4fb-9891-4610-82a5-e6e79602ff3f" (UID: "29a7d4fb-9891-4610-82a5-e6e79602ff3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:17.481027 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.481008    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a7d4fb-9891-4610-82a5-e6e79602ff3f-kube-api-access-xrrhd" pod "29a7d4fb-9891-4610-82a5-e6e79602ff3f" (UID: "29a7d4fb-9891-4610-82a5-e6e79602ff3f"). InnerVolumeSpecName "kube-api-access-xrrhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:17.546994 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:17.546958    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:45:17.547860 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.547818    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"]
Nov 01 06:45:17.583440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.580626    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrrhd\" (UniqueName: \"kubernetes.io/projected/29a7d4fb-9891-4610-82a5-e6e79602ff3f-kube-api-access-xrrhd\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:17.583440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.580655    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a7d4fb-9891-4610-82a5-e6e79602ff3f-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:17.709922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.709895    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:17.782252 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.782167    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dacc97f-943c-45b4-a8d7-c2805a393c01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dacc97f-943c-45b4-a8d7-c2805a393c01-maas-api-service-ca\") pod \"5dacc97f-943c-45b4-a8d7-c2805a393c01\" (UID: \"5dacc97f-943c-45b4-a8d7-c2805a393c01\") "
Nov 01 06:45:17.782406 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.782271    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dacc97f-943c-45b4-a8d7-c2805a393c01-kube-api-access-qrcsq\" (UniqueName: \"kubernetes.io/projected/5dacc97f-943c-45b4-a8d7-c2805a393c01-kube-api-access-qrcsq\") pod \"5dacc97f-943c-45b4-a8d7-c2805a393c01\" (UID: \"5dacc97f-943c-45b4-a8d7-c2805a393c01\") "
Nov 01 06:45:17.782605 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.782580    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dacc97f-943c-45b4-a8d7-c2805a393c01-maas-api-service-ca" pod "5dacc97f-943c-45b4-a8d7-c2805a393c01" (UID: "5dacc97f-943c-45b4-a8d7-c2805a393c01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:17.784524 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.784502    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dacc97f-943c-45b4-a8d7-c2805a393c01-kube-api-access-qrcsq" pod "5dacc97f-943c-45b4-a8d7-c2805a393c01" (UID: "5dacc97f-943c-45b4-a8d7-c2805a393c01"). InnerVolumeSpecName "kube-api-access-qrcsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:17.883405 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.883372    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dacc97f-943c-45b4-a8d7-c2805a393c01-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:17.883405 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:17.883399    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrcsq\" (UniqueName: \"kubernetes.io/projected/5dacc97f-943c-45b4-a8d7-c2805a393c01-kube-api-access-qrcsq\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:18.213612 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.213587    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"
Nov 01 06:45:18.214097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.213633    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb" event={"ID":"5dacc97f-943c-45b4-a8d7-c2805a393c01","Type":"ContainerDied","Data":"97df27e59bf8a5bab0f17635f0b133f7e067464313d9877ad53557e9832a75fa"}
Nov 01 06:45:18.214097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.213709    5510 scope.go:122] "RemoveContainer" containerID="cfcede506d317ff5d16de62dd5a012150a3835bcb4b94a16e5e2ba3109c5132c"
Nov 01 06:45:18.215112 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.215095    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"
Nov 01 06:45:18.215227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.215139    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx" event={"ID":"29a7d4fb-9891-4610-82a5-e6e79602ff3f","Type":"ContainerDied","Data":"709f5a8f6b05ab1c89d03e93921fbae91df56ffc88f13a5a8618b6738da999e1"}
Nov 01 06:45:18.219664 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.219637    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" event={"ID":"cad581a7-fb28-4443-8900-1b3083749f18","Type":"ContainerStarted","Data":"1527a77a9b1abad001b130f69532ee59ee4700c2e9da0ede2cfa45d41482c0be"}
Nov 01 06:45:18.225930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.225912    5510 scope.go:122] "RemoveContainer" containerID="b613f57b19174ec6cbd29532f341eab989f9b959e55f8772fd1e8439fa80012f"
Nov 01 06:45:18.228918 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.228898    5510 scope.go:122] "RemoveContainer" containerID="385e60f18a1bc23ed538478bbc5b918e7cda8ff39b39dd2c4de539c66aaf6cf0"
Nov 01 06:45:18.380426 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.380381    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"]
Nov 01 06:45:18.384079 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.384051    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f73e89-29891925-kgsdx"]
Nov 01 06:45:18.391490 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.391466    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:18.490207 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.490112    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"]
Nov 01 06:45:18.490474 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.490458    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54b12b6-9821-486a-ab48-84bbc3855ff4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54b12b6-9821-486a-ab48-84bbc3855ff4-maas-api-service-ca\") pod \"a54b12b6-9821-486a-ab48-84bbc3855ff4\" (UID: \"a54b12b6-9821-486a-ab48-84bbc3855ff4\") "
Nov 01 06:45:18.490557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.490542    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54b12b6-9821-486a-ab48-84bbc3855ff4-kube-api-access-xl4gb\" (UniqueName: \"kubernetes.io/projected/a54b12b6-9821-486a-ab48-84bbc3855ff4-kube-api-access-xl4gb\") pod \"a54b12b6-9821-486a-ab48-84bbc3855ff4\" (UID: \"a54b12b6-9821-486a-ab48-84bbc3855ff4\") "
Nov 01 06:45:18.490874 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.490852    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54b12b6-9821-486a-ab48-84bbc3855ff4-maas-api-service-ca" pod "a54b12b6-9821-486a-ab48-84bbc3855ff4" (UID: "a54b12b6-9821-486a-ab48-84bbc3855ff4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:18.491062 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.491040    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54b12b6-9821-486a-ab48-84bbc3855ff4-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:18.492857 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.492813    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7d25db-29891925-bb6cb"]
Nov 01 06:45:18.492945 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.492867    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54b12b6-9821-486a-ab48-84bbc3855ff4-kube-api-access-xl4gb" pod "a54b12b6-9821-486a-ab48-84bbc3855ff4" (UID: "a54b12b6-9821-486a-ab48-84bbc3855ff4"). InnerVolumeSpecName "kube-api-access-xl4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:18.592748 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:18.592714    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl4gb\" (UniqueName: \"kubernetes.io/projected/a54b12b6-9821-486a-ab48-84bbc3855ff4-kube-api-access-xl4gb\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:19.227186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.227155    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"
Nov 01 06:45:19.227186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.227165    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj" event={"ID":"a54b12b6-9821-486a-ab48-84bbc3855ff4","Type":"ContainerDied","Data":"21bf8706bc71418c678c4d7b9121ce5b7303b9596f147c2fb03ccc0ddbe0f3e7"}
Nov 01 06:45:19.227757 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.227275    5510 scope.go:122] "RemoveContainer" containerID="7b7942636df375b5e960b36e2456adf4b062171a7bf1f985f5f98cccc569aa27"
Nov 01 06:45:19.229258 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.229232    5510 generic.go:356] "Generic (PLEG): container finished" podID="cad581a7-fb28-4443-8900-1b3083749f18" containerID="1527a77a9b1abad001b130f69532ee59ee4700c2e9da0ede2cfa45d41482c0be" exitCode=60
Nov 01 06:45:19.236895 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.236811    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" path="/var/lib/kubelet/pods/29a7d4fb-9891-4610-82a5-e6e79602ff3f/volumes"
Nov 01 06:45:19.237334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.237315    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" path="/var/lib/kubelet/pods/5dacc97f-943c-45b4-a8d7-c2805a393c01/volumes"
Nov 01 06:45:19.237820 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.237801    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" event={"ID":"cad581a7-fb28-4443-8900-1b3083749f18","Type":"ContainerDied","Data":"1527a77a9b1abad001b130f69532ee59ee4700c2e9da0ede2cfa45d41482c0be"}
Nov 01 06:45:19.237929 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.237884    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" event={"ID":"0a81091f-96ea-49e6-899c-d64803b91306","Type":"ContainerStarted","Data":"7f5a9ca88e27fe09903e7dacd0f3291baafc82e933f211b6613d6442a15a22b1"}
Nov 01 06:45:19.238370 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.238348    5510 scope.go:122] "RemoveContainer" containerID="1527a77a9b1abad001b130f69532ee59ee4700c2e9da0ede2cfa45d41482c0be"
Nov 01 06:45:19.238564 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:19.238545    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95_odh-ai-gateway-infra(cad581a7-fb28-4443-8900-1b3083749f18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" podUID="cad581a7-fb28-4443-8900-1b3083749f18"
Nov 01 06:45:19.239325 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.239301    5510 scope.go:122] "RemoveContainer" containerID="86d19f886b4fd5ab88885c71387be06af34bc77b233b98b6acf3f96b16839eca"
Nov 01 06:45:19.613335 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.613244    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"]
Nov 01 06:45:19.619753 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.619726    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"]
Nov 01 06:45:19.625871 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:19.625850    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-14a6e4-29891925-m6vtj"]
Nov 01 06:45:20.243532 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.243503    5510 generic.go:356] "Generic (PLEG): container finished" podID="0a81091f-96ea-49e6-899c-d64803b91306" containerID="7f5a9ca88e27fe09903e7dacd0f3291baafc82e933f211b6613d6442a15a22b1" exitCode=60
Nov 01 06:45:20.244122 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.243564    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" event={"ID":"0a81091f-96ea-49e6-899c-d64803b91306","Type":"ContainerDied","Data":"7f5a9ca88e27fe09903e7dacd0f3291baafc82e933f211b6613d6442a15a22b1"}
Nov 01 06:45:20.244122 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.243625    5510 scope.go:122] "RemoveContainer" containerID="385e60f18a1bc23ed538478bbc5b918e7cda8ff39b39dd2c4de539c66aaf6cf0"
Nov 01 06:45:20.244122 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.244053    5510 scope.go:122] "RemoveContainer" containerID="7f5a9ca88e27fe09903e7dacd0f3291baafc82e933f211b6613d6442a15a22b1"
Nov 01 06:45:20.244414 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:20.244390    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891925-bcln5_odh-ai-gateway-infra(0a81091f-96ea-49e6-899c-d64803b91306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" podUID="0a81091f-96ea-49e6-899c-d64803b91306"
Nov 01 06:45:20.402967 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.402941    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"]
Nov 01 06:45:20.414358 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.414335    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:20.515941 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.515851    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad581a7-fb28-4443-8900-1b3083749f18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad581a7-fb28-4443-8900-1b3083749f18-maas-api-service-ca\") pod \"cad581a7-fb28-4443-8900-1b3083749f18\" (UID: \"cad581a7-fb28-4443-8900-1b3083749f18\") "
Nov 01 06:45:20.515941 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.515930    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad581a7-fb28-4443-8900-1b3083749f18-kube-api-access-hm5dg\" (UniqueName: \"kubernetes.io/projected/cad581a7-fb28-4443-8900-1b3083749f18-kube-api-access-hm5dg\") pod \"cad581a7-fb28-4443-8900-1b3083749f18\" (UID: \"cad581a7-fb28-4443-8900-1b3083749f18\") "
Nov 01 06:45:20.516242 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.516218    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad581a7-fb28-4443-8900-1b3083749f18-maas-api-service-ca" pod "cad581a7-fb28-4443-8900-1b3083749f18" (UID: "cad581a7-fb28-4443-8900-1b3083749f18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:20.518196 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.518174    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad581a7-fb28-4443-8900-1b3083749f18-kube-api-access-hm5dg" pod "cad581a7-fb28-4443-8900-1b3083749f18" (UID: "cad581a7-fb28-4443-8900-1b3083749f18"). InnerVolumeSpecName "kube-api-access-hm5dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:20.617687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.617647    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad581a7-fb28-4443-8900-1b3083749f18-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:20.617687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:20.617675    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm5dg\" (UniqueName: \"kubernetes.io/projected/cad581a7-fb28-4443-8900-1b3083749f18-kube-api-access-hm5dg\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:21.244695 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.244654    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" path="/var/lib/kubelet/pods/a54b12b6-9821-486a-ab48-84bbc3855ff4/volumes"
Nov 01 06:45:21.254384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.254361    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"
Nov 01 06:45:21.254384 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.254375    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95" event={"ID":"cad581a7-fb28-4443-8900-1b3083749f18","Type":"ContainerDied","Data":"b308bf010b5929a592c0e2cfbf3c9133ba61b82379dd5faf095bc6a81113c4f3"}
Nov 01 06:45:21.254591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.254434    5510 scope.go:122] "RemoveContainer" containerID="1527a77a9b1abad001b130f69532ee59ee4700c2e9da0ede2cfa45d41482c0be"
Nov 01 06:45:21.410266 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.410241    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:21.525899 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.525804    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a81091f-96ea-49e6-899c-d64803b91306-kube-api-access-2f92m\" (UniqueName: \"kubernetes.io/projected/0a81091f-96ea-49e6-899c-d64803b91306-kube-api-access-2f92m\") pod \"0a81091f-96ea-49e6-899c-d64803b91306\" (UID: \"0a81091f-96ea-49e6-899c-d64803b91306\") "
Nov 01 06:45:21.525899 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.525882    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a81091f-96ea-49e6-899c-d64803b91306-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81091f-96ea-49e6-899c-d64803b91306-maas-api-service-ca\") pod \"0a81091f-96ea-49e6-899c-d64803b91306\" (UID: \"0a81091f-96ea-49e6-899c-d64803b91306\") "
Nov 01 06:45:21.526208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.526186    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a81091f-96ea-49e6-899c-d64803b91306-maas-api-service-ca" pod "0a81091f-96ea-49e6-899c-d64803b91306" (UID: "0a81091f-96ea-49e6-899c-d64803b91306"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:21.526314 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.526297    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81091f-96ea-49e6-899c-d64803b91306-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:21.528904 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.528877    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a81091f-96ea-49e6-899c-d64803b91306-kube-api-access-2f92m" pod "0a81091f-96ea-49e6-899c-d64803b91306" (UID: "0a81091f-96ea-49e6-899c-d64803b91306"). InnerVolumeSpecName "kube-api-access-2f92m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:21.532170 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.532139    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"]
Nov 01 06:45:21.535777 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.535751    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8f45a2-29891925-d2b95"]
Nov 01 06:45:21.627531 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:21.627495    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f92m\" (UniqueName: \"kubernetes.io/projected/0a81091f-96ea-49e6-899c-d64803b91306-kube-api-access-2f92m\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:22.264692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:22.264668    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"
Nov 01 06:45:22.265218 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:22.264710    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5" event={"ID":"0a81091f-96ea-49e6-899c-d64803b91306","Type":"ContainerDied","Data":"04b8ebf844ccd4af5bf9a7db199cc6383625fc7228e54511f1d8abd26e9fef77"}
Nov 01 06:45:22.265218 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:22.264764    5510 scope.go:122] "RemoveContainer" containerID="7f5a9ca88e27fe09903e7dacd0f3291baafc82e933f211b6613d6442a15a22b1"
Nov 01 06:45:22.427386 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:22.427347    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"]
Nov 01 06:45:22.433325 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:22.433298    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-bcln5"]
Nov 01 06:45:23.236631 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:23.236597    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a81091f-96ea-49e6-899c-d64803b91306" path="/var/lib/kubelet/pods/0a81091f-96ea-49e6-899c-d64803b91306/volumes"
Nov 01 06:45:23.237035 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:23.237021    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad581a7-fb28-4443-8900-1b3083749f18" path="/var/lib/kubelet/pods/cad581a7-fb28-4443-8900-1b3083749f18/volumes"
Nov 01 06:45:47.545917 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:45:47.545884    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:45:52.256246 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:52.256212    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"]
Nov 01 06:45:52.305650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:52.305618    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"]
Nov 01 06:45:55.812533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:45:55.812502    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:46:02.562870 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.562801    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"]
Nov 01 06:46:02.563569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563547    5510 cpu_manager.go:414] "containerMap: removing container" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:02.563671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563570    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:02.563671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563591    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:02.563671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563598    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:02.563671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563622    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:02.563671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563630    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:02.563671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563647    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:02.563671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563654    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:02.563671 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563671    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563678    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563689    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563696    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563708    5510 cpu_manager.go:414] "containerMap: removing container" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563716    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563727    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563735    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563749    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563756    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563767    5510 cpu_manager.go:414] "containerMap: removing container" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563774    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563786    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563792    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563804    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563812    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563872    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.563879    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564019    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564036    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564049    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564061    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564070    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564078    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a81091f-96ea-49e6-899c-d64803b91306" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564089    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564099    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564107    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564119    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:02.564126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.564129    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:02.568171 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.568152    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.668780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.668750    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d2260d-openshift-default-dockercfg-6tk65\""
Nov 01 06:46:02.700876 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.700809    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"]
Nov 01 06:46:02.725102 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725065    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-credential-socket\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.725272 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725115    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-data\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.725272 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725266    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-istio-token\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.725359 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725306    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-socket\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.725359 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725346    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c10027c-e632-466c-b086-6fef6fed4292-istio-podinfo\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.725433 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725407    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-envoy\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.725475 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725444    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8nlm\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-kube-api-access-c8nlm\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.725518 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725479    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c10027c-e632-466c-b086-6fef6fed4292-istiod-ca-cert\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.725553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.725525    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-certs\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827299 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827210    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-envoy\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827299 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827257    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8nlm\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-kube-api-access-c8nlm\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827299 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827282    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c10027c-e632-466c-b086-6fef6fed4292-istiod-ca-cert\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827327    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-certs\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827385    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-credential-socket\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827415    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-data\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827501    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-istio-token\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827538    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-socket\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827591 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827581    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c10027c-e632-466c-b086-6fef6fed4292-istio-podinfo\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.827949 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827783    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-certs\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.828015 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.827948    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-credential-socket\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.828015 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.828003    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-data\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.828174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.828147    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-socket\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.828315 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.828297    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c10027c-e632-466c-b086-6fef6fed4292-istiod-ca-cert\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.829784 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.829762    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-envoy\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.830377 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.830361    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c10027c-e632-466c-b086-6fef6fed4292-istio-podinfo\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.897873 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.894434    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-istio-token\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:02.910725 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:02.910628    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8nlm\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-kube-api-access-c8nlm\") pod \"e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") " pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:03.181722 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:03.181634    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:03.620073 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:03.620044    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"]
Nov 01 06:46:03.625537 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:46:03.625509    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c10027c_e632_466c_b086_6fef6fed4292.slice/crio-1c4ea7bed51cc9658b50a14e6588ba63f28d93aaceab4dbe5989f989c63b5f12 WatchSource:0}: Error finding container 1c4ea7bed51cc9658b50a14e6588ba63f28d93aaceab4dbe5989f989c63b5f12: Status 404 returned error can't find the container with id 1c4ea7bed51cc9658b50a14e6588ba63f28d93aaceab4dbe5989f989c63b5f12
Nov 01 06:46:03.627729 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:03.627696    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:03.627801 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:03.627762    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:03.627801 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:03.627793    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:04.519298 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:04.519257    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" event={"ID":"0c10027c-e632-466c-b086-6fef6fed4292","Type":"ContainerStarted","Data":"a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532"}
Nov 01 06:46:04.519430 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:04.519343    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" event={"ID":"0c10027c-e632-466c-b086-6fef6fed4292","Type":"ContainerStarted","Data":"1c4ea7bed51cc9658b50a14e6588ba63f28d93aaceab4dbe5989f989c63b5f12"}
Nov 01 06:46:04.639479 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:04.639430    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" podStartSLOduration=2.639415563 podStartE2EDuration="2.639415563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:04.63790639 +0000 UTC m=+3601.515879802" watchObservedRunningTime="2026-11-01 06:46:04.639415563 +0000 UTC m=+3601.517388974"
Nov 01 06:46:05.182729 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:05.182698    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:05.189077 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:05.189053    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:05.462407 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:05.462375    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"]
Nov 01 06:46:05.526357 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:05.526324    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:05.527288 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:05.527265    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:46:05.573777 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:05.573743    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"]
Nov 01 06:46:07.277634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.277587    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd" podUID="84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" containerName="main" containerID="cri-o://0fcd12be4cfb965aa9cae68c78fe780687b1109cf299ac4d71ef54e5e6e1c6f3" gracePeriod=45
Nov 01 06:46:07.325553 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.325501    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv" podUID="bb5db556-563e-43f5-ab8a-30d2b3df2254" containerName="main" containerID="cri-o://5c1f51102210ba6ad04bbbb6d52fad211237a6e641a52afc1ae26db748a61564" gracePeriod=45
Nov 01 06:46:07.335159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.335130    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd" podUID="84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused"
Nov 01 06:46:07.540794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.540718    5510 generic.go:356] "Generic (PLEG): container finished" podID="84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" containerID="0fcd12be4cfb965aa9cae68c78fe780687b1109cf299ac4d71ef54e5e6e1c6f3" exitCode=0
Nov 01 06:46:07.540794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.540766    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd" event={"ID":"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3","Type":"ContainerDied","Data":"0fcd12be4cfb965aa9cae68c78fe780687b1109cf299ac4d71ef54e5e6e1c6f3"}
Nov 01 06:46:07.542604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.542579    5510 generic.go:356] "Generic (PLEG): container finished" podID="bb5db556-563e-43f5-ab8a-30d2b3df2254" containerID="5c1f51102210ba6ad04bbbb6d52fad211237a6e641a52afc1ae26db748a61564" exitCode=0
Nov 01 06:46:07.542721 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.542666    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv" event={"ID":"bb5db556-563e-43f5-ab8a-30d2b3df2254","Type":"ContainerDied","Data":"5c1f51102210ba6ad04bbbb6d52fad211237a6e641a52afc1ae26db748a61564"}
Nov 01 06:46:07.591593 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.591563    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:46:07.650078 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.650053    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 06:46:07.702643 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.702608    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-home\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-home\") pod \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") "
Nov 01 06:46:07.702796 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.702757    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-kube-api-access-7q54n\" (UniqueName: \"kubernetes.io/projected/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-kube-api-access-7q54n\") pod \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") "
Nov 01 06:46:07.702906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.702802    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-dshm\") pod \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") "
Nov 01 06:46:07.702967 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.702933    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-model-cache\") pod \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") "
Nov 01 06:46:07.703021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.702971    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tmp-dir\") pod \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") "
Nov 01 06:46:07.703021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.702998    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tls-certs\") pod \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\" (UID: \"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3\") "
Nov 01 06:46:07.703130 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.702932    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-home" pod "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" (UID: "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.703604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.703317    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-model-cache" pod "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" (UID: "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.703604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.703558    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-model-cache\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.703604 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.703577    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-home\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.703885 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.703811    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tmp-dir" pod "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" (UID: "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.705655 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.705631    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-dshm" pod "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" (UID: "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.705780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.705748    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-kube-api-access-7q54n" pod "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" (UID: "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3"). InnerVolumeSpecName "kube-api-access-7q54n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:07.705908 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.705886    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tls-certs" pod "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" (UID: "84a010d6-0c1a-4a15-8af7-239ab2ecf8f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:46:07.804981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.804891    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-home\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-home\") pod \"bb5db556-563e-43f5-ab8a-30d2b3df2254\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") "
Nov 01 06:46:07.804981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.804972    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5db556-563e-43f5-ab8a-30d2b3df2254-kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/bb5db556-563e-43f5-ab8a-30d2b3df2254-kube-api-access-7jtj9\") pod \"bb5db556-563e-43f5-ab8a-30d2b3df2254\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") "
Nov 01 06:46:07.805208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805017    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-tmp-dir\") pod \"bb5db556-563e-43f5-ab8a-30d2b3df2254\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") "
Nov 01 06:46:07.805208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805050    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-model-cache\") pod \"bb5db556-563e-43f5-ab8a-30d2b3df2254\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") "
Nov 01 06:46:07.805208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805127    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-dshm\") pod \"bb5db556-563e-43f5-ab8a-30d2b3df2254\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") "
Nov 01 06:46:07.805208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805164    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5db556-563e-43f5-ab8a-30d2b3df2254-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5db556-563e-43f5-ab8a-30d2b3df2254-tls-certs\") pod \"bb5db556-563e-43f5-ab8a-30d2b3df2254\" (UID: \"bb5db556-563e-43f5-ab8a-30d2b3df2254\") "
Nov 01 06:46:07.805208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805188    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-home" pod "bb5db556-563e-43f5-ab8a-30d2b3df2254" (UID: "bb5db556-563e-43f5-ab8a-30d2b3df2254"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.805470 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805311    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-tmp-dir" pod "bb5db556-563e-43f5-ab8a-30d2b3df2254" (UID: "bb5db556-563e-43f5-ab8a-30d2b3df2254"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.805470 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805362    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-model-cache" pod "bb5db556-563e-43f5-ab8a-30d2b3df2254" (UID: "bb5db556-563e-43f5-ab8a-30d2b3df2254"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.805711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805685    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q54n\" (UniqueName: \"kubernetes.io/projected/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-kube-api-access-7q54n\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.805711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805713    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-dshm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.806134 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805728    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tmp-dir\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.806134 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805742    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3-tls-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.806134 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805755    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-home\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.806134 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805767    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-tmp-dir\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.806134 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.805781    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-model-cache\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.807509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.807484    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5db556-563e-43f5-ab8a-30d2b3df2254-kube-api-access-7jtj9" pod "bb5db556-563e-43f5-ab8a-30d2b3df2254" (UID: "bb5db556-563e-43f5-ab8a-30d2b3df2254"). InnerVolumeSpecName "kube-api-access-7jtj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:07.807659 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.807633    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5db556-563e-43f5-ab8a-30d2b3df2254-tls-certs" pod "bb5db556-563e-43f5-ab8a-30d2b3df2254" (UID: "bb5db556-563e-43f5-ab8a-30d2b3df2254"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:46:07.807774 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.807714    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-dshm" pod "bb5db556-563e-43f5-ab8a-30d2b3df2254" (UID: "bb5db556-563e-43f5-ab8a-30d2b3df2254"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.906838 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.906803    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/bb5db556-563e-43f5-ab8a-30d2b3df2254-kube-api-access-7jtj9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.907001 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.906863    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb5db556-563e-43f5-ab8a-30d2b3df2254-dshm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.907001 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:07.906874    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5db556-563e-43f5-ab8a-30d2b3df2254-tls-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:08.550508 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.550415    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd" event={"ID":"84a010d6-0c1a-4a15-8af7-239ab2ecf8f3","Type":"ContainerDied","Data":"110b4a07b0a66252b67f67597867d10a413e33e6f4a4dc9d3917cda39f19eab5"}
Nov 01 06:46:08.550508 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.550446    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"
Nov 01 06:46:08.550508 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.550464    5510 scope.go:122] "RemoveContainer" containerID="0fcd12be4cfb965aa9cae68c78fe780687b1109cf299ac4d71ef54e5e6e1c6f3"
Nov 01 06:46:08.552193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.552154    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"
Nov 01 06:46:08.552193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.552188    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv" event={"ID":"bb5db556-563e-43f5-ab8a-30d2b3df2254","Type":"ContainerDied","Data":"1de5cacc1845d6aec1191937cef6a57e39cbc0e860d1fed170db32c897a45aa4"}
Nov 01 06:46:08.561354 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.561340    5510 scope.go:122] "RemoveContainer" containerID="5c1f51102210ba6ad04bbbb6d52fad211237a6e641a52afc1ae26db748a61564"
Nov 01 06:46:08.722742 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.722710    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"]
Nov 01 06:46:08.771861 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.771816    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-w4-f73e89-kserve-d6669785c-n9tkd"]
Nov 01 06:46:08.880385 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.880312    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"]
Nov 01 06:46:08.884108 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:08.884085    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/premium-w4-f73e89-kserve-54578f8898-vzxkv"]
Nov 01 06:46:09.236428 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:09.236394    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" path="/var/lib/kubelet/pods/84a010d6-0c1a-4a15-8af7-239ab2ecf8f3/volumes"
Nov 01 06:46:09.236805 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:09.236791    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5db556-563e-43f5-ab8a-30d2b3df2254" path="/var/lib/kubelet/pods/bb5db556-563e-43f5-ab8a-30d2b3df2254/volumes"
Nov 01 06:46:10.463502 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:10.463465    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp" podUID="8d987f61-f6ba-4950-9ba2-7023096db483" containerName="payload-processing" containerID="cri-o://25db57e67ed6a9191c1500bccc4c6074e099aed23e5d5209c9f141764fdad3da" gracePeriod=25
Nov 01 06:46:10.574442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:10.574403    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh" podUID="5e7eaa36-4938-4d11-8db2-304a18f0de38" containerName="payload-pre-processing" containerID="cri-o://0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f" gracePeriod=25
Nov 01 06:46:11.214635 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:11.214580    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp" podUID="8d987f61-f6ba-4950-9ba2-7023096db483" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:46:11.214635 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:11.214618    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh" podUID="5e7eaa36-4938-4d11-8db2-304a18f0de38" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:46:17.547134 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:46:17.547059    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:46:21.214077 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.214037    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp" podUID="8d987f61-f6ba-4950-9ba2-7023096db483" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:46:21.214537 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.214039    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh" podUID="5e7eaa36-4938-4d11-8db2-304a18f0de38" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:46:21.309495 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.309463    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"]
Nov 01 06:46:21.310587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.310557    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:21.310759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.310742    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:21.311062 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.310909    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:21.311301 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.311237    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:21.312304 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.312287    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bb5db556-563e-43f5-ab8a-30d2b3df2254" containerName="main"
Nov 01 06:46:21.312446 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.312398    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5db556-563e-43f5-ab8a-30d2b3df2254" containerName="main"
Nov 01 06:46:21.313607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.312560    5510 cpu_manager.go:414] "containerMap: removing container" podUID="84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" containerName="main"
Nov 01 06:46:21.313751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.313735    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" containerName="main"
Nov 01 06:46:21.314027 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.314010    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a010d6-0c1a-4a15-8af7-239ab2ecf8f3" containerName="main"
Nov 01 06:46:21.314150 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.314138    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54b12b6-9821-486a-ab48-84bbc3855ff4" containerName="cleanup"
Nov 01 06:46:21.314243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.314233    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad581a7-fb28-4443-8900-1b3083749f18" containerName="cleanup"
Nov 01 06:46:21.314333 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.314323    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a7d4fb-9891-4610-82a5-e6e79602ff3f" containerName="cleanup"
Nov 01 06:46:21.314420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.314410    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dacc97f-943c-45b4-a8d7-c2805a393c01" containerName="cleanup"
Nov 01 06:46:21.314503 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.314493    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5db556-563e-43f5-ab8a-30d2b3df2254" containerName="main"
Nov 01 06:46:21.323943 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.323922    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:21.368059 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.368024    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f179c728-3846-4be0-b477-6dc3d9b7ec46-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8\" (UID: \"f179c728-3846-4be0-b477-6dc3d9b7ec46\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:21.368246 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.368095    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgx92\" (UniqueName: \"kubernetes.io/projected/f179c728-3846-4be0-b477-6dc3d9b7ec46-kube-api-access-wgx92\") pod \"payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8\" (UID: \"f179c728-3846-4be0-b477-6dc3d9b7ec46\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:21.373350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.373287    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"]
Nov 01 06:46:21.419981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.419951    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d2260d-dockercfg-qdbzx\""
Nov 01 06:46:21.419981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.419977    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d2260d\""
Nov 01 06:46:21.469409 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.469342    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"]
Nov 01 06:46:21.469907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.469883    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f179c728-3846-4be0-b477-6dc3d9b7ec46-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8\" (UID: \"f179c728-3846-4be0-b477-6dc3d9b7ec46\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:21.469987 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.469937    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgx92\" (UniqueName: \"kubernetes.io/projected/f179c728-3846-4be0-b477-6dc3d9b7ec46-kube-api-access-wgx92\") pod \"payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8\" (UID: \"f179c728-3846-4be0-b477-6dc3d9b7ec46\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:21.470596 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.470577    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f179c728-3846-4be0-b477-6dc3d9b7ec46-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8\" (UID: \"f179c728-3846-4be0-b477-6dc3d9b7ec46\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:21.473334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.473321    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:21.530209 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.530177    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"]
Nov 01 06:46:21.530917 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.530895    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgx92\" (UniqueName: \"kubernetes.io/projected/f179c728-3846-4be0-b477-6dc3d9b7ec46-kube-api-access-wgx92\") pod \"payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8\" (UID: \"f179c728-3846-4be0-b477-6dc3d9b7ec46\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:21.570774 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.570734    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a9a2-4627-4e04-af0a-ddb34fc205be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4\" (UID: \"6738a9a2-4627-4e04-af0a-ddb34fc205be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:21.570956 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.570818    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fm5k\" (UniqueName: \"kubernetes.io/projected/6738a9a2-4627-4e04-af0a-ddb34fc205be-kube-api-access-8fm5k\") pod \"payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4\" (UID: \"6738a9a2-4627-4e04-af0a-ddb34fc205be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:21.663055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.663022    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:21.671502 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.671475    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fm5k\" (UniqueName: \"kubernetes.io/projected/6738a9a2-4627-4e04-af0a-ddb34fc205be-kube-api-access-8fm5k\") pod \"payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4\" (UID: \"6738a9a2-4627-4e04-af0a-ddb34fc205be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:21.671614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.671602    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a9a2-4627-4e04-af0a-ddb34fc205be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4\" (UID: \"6738a9a2-4627-4e04-af0a-ddb34fc205be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:21.672224 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.672205    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a9a2-4627-4e04-af0a-ddb34fc205be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4\" (UID: \"6738a9a2-4627-4e04-af0a-ddb34fc205be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:21.749443 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.749410    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fm5k\" (UniqueName: \"kubernetes.io/projected/6738a9a2-4627-4e04-af0a-ddb34fc205be-kube-api-access-8fm5k\") pod \"payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4\" (UID: \"6738a9a2-4627-4e04-af0a-ddb34fc205be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:21.788352 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:21.788321    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:22.075363 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.075334    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"]
Nov 01 06:46:22.179218 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.179190    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"]
Nov 01 06:46:22.180373 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:46:22.180346    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6738a9a2_4627_4e04_af0a_ddb34fc205be.slice/crio-598b9fe524bf91d859e5f155c351dc2dd88792104b7f38266af61f31477656b8 WatchSource:0}: Error finding container 598b9fe524bf91d859e5f155c351dc2dd88792104b7f38266af61f31477656b8: Status 404 returned error can't find the container with id 598b9fe524bf91d859e5f155c351dc2dd88792104b7f38266af61f31477656b8
Nov 01 06:46:22.635529 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.635496    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8" event={"ID":"f179c728-3846-4be0-b477-6dc3d9b7ec46","Type":"ContainerStarted","Data":"f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2"}
Nov 01 06:46:22.636039 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.635559    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8" event={"ID":"f179c728-3846-4be0-b477-6dc3d9b7ec46","Type":"ContainerStarted","Data":"25715929f2f4342b4603dfa9cb4418491ef47a483148316c8b188da0b4c23c8b"}
Nov 01 06:46:22.636039 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.635589    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:22.637445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.637413    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4" event={"ID":"6738a9a2-4627-4e04-af0a-ddb34fc205be","Type":"ContainerStarted","Data":"e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088"}
Nov 01 06:46:22.637574 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.637468    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4" event={"ID":"6738a9a2-4627-4e04-af0a-ddb34fc205be","Type":"ContainerStarted","Data":"598b9fe524bf91d859e5f155c351dc2dd88792104b7f38266af61f31477656b8"}
Nov 01 06:46:22.637574 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.637500    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:22.752498 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.752439    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8" podStartSLOduration=1.7524250380000002 podStartE2EDuration="1.752425038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:22.74955729 +0000 UTC m=+3619.627530740" watchObservedRunningTime="2026-11-01 06:46:22.752425038 +0000 UTC m=+3619.630398448"
Nov 01 06:46:22.880221 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:22.880153    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4" podStartSLOduration=1.880129929 podStartE2EDuration="1.880129929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:22.874261713 +0000 UTC m=+3619.752235124" watchObservedRunningTime="2026-11-01 06:46:22.880129929 +0000 UTC m=+3619.758103343"
Nov 01 06:46:30.689621 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.689582    5510 generic.go:356] "Generic (PLEG): container finished" podID="8d987f61-f6ba-4950-9ba2-7023096db483" containerID="25db57e67ed6a9191c1500bccc4c6074e099aed23e5d5209c9f141764fdad3da" exitCode=0
Nov 01 06:46:30.690045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.689652    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp" event={"ID":"8d987f61-f6ba-4950-9ba2-7023096db483","Type":"ContainerDied","Data":"25db57e67ed6a9191c1500bccc4c6074e099aed23e5d5209c9f141764fdad3da"}
Nov 01 06:46:30.721821 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.721755    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 06:46:30.805158 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.805127    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d987f61-f6ba-4950-9ba2-7023096db483-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d987f61-f6ba-4950-9ba2-7023096db483-plugins-config-volume\") pod \"8d987f61-f6ba-4950-9ba2-7023096db483\" (UID: \"8d987f61-f6ba-4950-9ba2-7023096db483\") "
Nov 01 06:46:30.805371 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.805357    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d987f61-f6ba-4950-9ba2-7023096db483-kube-api-access-2nkgs\" (UniqueName: \"kubernetes.io/projected/8d987f61-f6ba-4950-9ba2-7023096db483-kube-api-access-2nkgs\") pod \"8d987f61-f6ba-4950-9ba2-7023096db483\" (UID: \"8d987f61-f6ba-4950-9ba2-7023096db483\") "
Nov 01 06:46:30.805545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.805523    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d987f61-f6ba-4950-9ba2-7023096db483-plugins-config-volume" pod "8d987f61-f6ba-4950-9ba2-7023096db483" (UID: "8d987f61-f6ba-4950-9ba2-7023096db483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:46:30.805670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.805654    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d987f61-f6ba-4950-9ba2-7023096db483-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:30.808164 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.808135    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d987f61-f6ba-4950-9ba2-7023096db483-kube-api-access-2nkgs" pod "8d987f61-f6ba-4950-9ba2-7023096db483" (UID: "8d987f61-f6ba-4950-9ba2-7023096db483"). InnerVolumeSpecName "kube-api-access-2nkgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:30.837807 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.837781    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 06:46:30.906453 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:30.906421    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nkgs\" (UniqueName: \"kubernetes.io/projected/8d987f61-f6ba-4950-9ba2-7023096db483-kube-api-access-2nkgs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:31.007248 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.007172    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7eaa36-4938-4d11-8db2-304a18f0de38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7eaa36-4938-4d11-8db2-304a18f0de38-plugins-config-volume\") pod \"5e7eaa36-4938-4d11-8db2-304a18f0de38\" (UID: \"5e7eaa36-4938-4d11-8db2-304a18f0de38\") "
Nov 01 06:46:31.007395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.007275    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7eaa36-4938-4d11-8db2-304a18f0de38-kube-api-access-clrnj\" (UniqueName: \"kubernetes.io/projected/5e7eaa36-4938-4d11-8db2-304a18f0de38-kube-api-access-clrnj\") pod \"5e7eaa36-4938-4d11-8db2-304a18f0de38\" (UID: \"5e7eaa36-4938-4d11-8db2-304a18f0de38\") "
Nov 01 06:46:31.007540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.007516    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7eaa36-4938-4d11-8db2-304a18f0de38-plugins-config-volume" pod "5e7eaa36-4938-4d11-8db2-304a18f0de38" (UID: "5e7eaa36-4938-4d11-8db2-304a18f0de38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:46:31.009431 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.009405    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7eaa36-4938-4d11-8db2-304a18f0de38-kube-api-access-clrnj" pod "5e7eaa36-4938-4d11-8db2-304a18f0de38" (UID: "5e7eaa36-4938-4d11-8db2-304a18f0de38"). InnerVolumeSpecName "kube-api-access-clrnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:31.109202 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.109172    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7eaa36-4938-4d11-8db2-304a18f0de38-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:31.109202 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.109199    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrnj\" (UniqueName: \"kubernetes.io/projected/5e7eaa36-4938-4d11-8db2-304a18f0de38-kube-api-access-clrnj\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:31.697773 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.697731    5510 generic.go:356] "Generic (PLEG): container finished" podID="5e7eaa36-4938-4d11-8db2-304a18f0de38" containerID="0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f" exitCode=0
Nov 01 06:46:31.698261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.697793    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"
Nov 01 06:46:31.698261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.697814    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh" event={"ID":"5e7eaa36-4938-4d11-8db2-304a18f0de38","Type":"ContainerDied","Data":"0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f"}
Nov 01 06:46:31.698261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.697890    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh" event={"ID":"5e7eaa36-4938-4d11-8db2-304a18f0de38","Type":"ContainerDied","Data":"6ee8c574dc8007c89cd86a4e4b26af408ef9abfc834f481259f4fc9b6a542f35"}
Nov 01 06:46:31.698261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.697920    5510 scope.go:122] "RemoveContainer" containerID="0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f"
Nov 01 06:46:31.699930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.699463    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"
Nov 01 06:46:31.699930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.699491    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp" event={"ID":"8d987f61-f6ba-4950-9ba2-7023096db483","Type":"ContainerDied","Data":"d775d30c6111c6d6798e0f891d385a506054f1c878f9d740b1a3c32282a758cd"}
Nov 01 06:46:31.707636 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.707621    5510 scope.go:122] "RemoveContainer" containerID="0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f"
Nov 01 06:46:31.707999 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:46:31.707981    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f\": container with ID starting with 0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f not found: ID does not exist" containerID="0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f"
Nov 01 06:46:31.708060 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.708006    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f"} err="failed to get container status \"0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f\": rpc error: code = NotFound desc = could not find container \"0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f\": container with ID starting with 0d39d889fc1a69fa649f582c6646aeb2c8bf0e63cc2dd8f328e9d18e07d1fb3f not found: ID does not exist"
Nov 01 06:46:31.708060 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.708023    5510 scope.go:122] "RemoveContainer" containerID="25db57e67ed6a9191c1500bccc4c6074e099aed23e5d5209c9f141764fdad3da"
Nov 01 06:46:31.873345 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.873255    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"]
Nov 01 06:46:31.910450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:31.910418    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f73e89-7984d89bcf-26fxp"]
Nov 01 06:46:32.023713 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:32.023682    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"]
Nov 01 06:46:32.040227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:32.040198    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f73e89-fbbcc795f-s8fqh"]
Nov 01 06:46:33.236368 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:33.236339    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7eaa36-4938-4d11-8db2-304a18f0de38" path="/var/lib/kubelet/pods/5e7eaa36-4938-4d11-8db2-304a18f0de38/volumes"
Nov 01 06:46:33.236737 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:33.236719    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d987f61-f6ba-4950-9ba2-7023096db483" path="/var/lib/kubelet/pods/8d987f61-f6ba-4950-9ba2-7023096db483/volumes"
Nov 01 06:46:33.647906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:33.647812    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:46:33.647906 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:33.647901    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:46:47.544464 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:46:47.544427    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:46:49.834280 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:46:49.834250    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:47:17.545622 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:47:17.545584    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:47:33.625811 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:33.625772    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:47:44.149311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:44.149278    5510 generic.go:356] "Generic (PLEG): container finished" podID="0c10027c-e632-466c-b086-6fef6fed4292" containerID="a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532" exitCode=0
Nov 01 06:47:44.149747 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:44.149358    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" event={"ID":"0c10027c-e632-466c-b086-6fef6fed4292","Type":"ContainerDied","Data":"a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532"}
Nov 01 06:47:44.149791 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:44.149767    5510 scope.go:122] "RemoveContainer" containerID="a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532"
Nov 01 06:47:44.151202 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:44.151176    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:44.151257 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:44.151245    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:44.151298 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:44.151276    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:45.157385 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:45.157350    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" event={"ID":"0c10027c-e632-466c-b086-6fef6fed4292","Type":"ContainerStarted","Data":"0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8"}
Nov 01 06:47:45.190090 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:45.189997    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:47:45.190696 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:45.190665    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 06:47:45.190997 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:45.190968    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 06:47:46.188214 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:46.188181    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 06:47:46.188809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:46.188778    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 06:47:47.190971 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:47.190932    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 06:47:47.191567 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:47.191530    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 06:47:47.544885 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:47:47.544819    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:47:48.183218 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:48.183183    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 06:47:48.183511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:48.183481    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 06:47:49.193858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:49.191210    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 06:47:49.193858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:49.191272    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 06:47:50.182605 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:50.182562    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 06:47:50.182794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:50.182629    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 06:47:50.526900 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:50.526869    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:47:51.187186 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:51.187155    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:47:51.216536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:51.216509    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:47:55.762287 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:55.762249    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"]
Nov 01 06:47:55.869874 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:55.867742    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"]
Nov 01 06:47:55.944854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:47:55.944053    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:48:00.762606 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:00.762543    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg" podUID="0bd3b495-c7e6-41e7-b06d-1014e71d0585" containerName="payload-processing" containerID="cri-o://48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407" gracePeriod=25
Nov 01 06:48:00.868633 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:00.868598    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd" podUID="72f6130c-64f5-4dbc-9553-0cc11c35edd8" containerName="payload-pre-processing" containerID="cri-o://ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6" gracePeriod=25
Nov 01 06:48:09.863437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:09.863399    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg" podUID="0bd3b495-c7e6-41e7-b06d-1014e71d0585" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:48:09.863889 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:09.863400    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd" podUID="72f6130c-64f5-4dbc-9553-0cc11c35edd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:48:17.546389 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:48:17.546343    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:48:19.862715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:19.862674    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg" podUID="0bd3b495-c7e6-41e7-b06d-1014e71d0585" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:48:19.862715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:19.862695    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd" podUID="72f6130c-64f5-4dbc-9553-0cc11c35edd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:48:21.034679 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.034616    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:48:21.128130 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.128105    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:48:21.207501 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.207471    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd3b495-c7e6-41e7-b06d-1014e71d0585-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3b495-c7e6-41e7-b06d-1014e71d0585-plugins-config-volume\") pod \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\" (UID: \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\") "
Nov 01 06:48:21.207687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.207557    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd3b495-c7e6-41e7-b06d-1014e71d0585-kube-api-access-2wwlv\" (UniqueName: \"kubernetes.io/projected/0bd3b495-c7e6-41e7-b06d-1014e71d0585-kube-api-access-2wwlv\") pod \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\" (UID: \"0bd3b495-c7e6-41e7-b06d-1014e71d0585\") "
Nov 01 06:48:21.207912 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.207886    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd3b495-c7e6-41e7-b06d-1014e71d0585-plugins-config-volume" pod "0bd3b495-c7e6-41e7-b06d-1014e71d0585" (UID: "0bd3b495-c7e6-41e7-b06d-1014e71d0585"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:21.209794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.209774    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd3b495-c7e6-41e7-b06d-1014e71d0585-kube-api-access-2wwlv" pod "0bd3b495-c7e6-41e7-b06d-1014e71d0585" (UID: "0bd3b495-c7e6-41e7-b06d-1014e71d0585"). InnerVolumeSpecName "kube-api-access-2wwlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:21.324714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.319781    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f6130c-64f5-4dbc-9553-0cc11c35edd8-kube-api-access-9w6f9\" (UniqueName: \"kubernetes.io/projected/72f6130c-64f5-4dbc-9553-0cc11c35edd8-kube-api-access-9w6f9\") pod \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\" (UID: \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\") "
Nov 01 06:48:21.324714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.319886    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f6130c-64f5-4dbc-9553-0cc11c35edd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6130c-64f5-4dbc-9553-0cc11c35edd8-plugins-config-volume\") pod \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\" (UID: \"72f6130c-64f5-4dbc-9553-0cc11c35edd8\") "
Nov 01 06:48:21.324714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.320498    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd3b495-c7e6-41e7-b06d-1014e71d0585-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:21.324714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.320516    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwlv\" (UniqueName: \"kubernetes.io/projected/0bd3b495-c7e6-41e7-b06d-1014e71d0585-kube-api-access-2wwlv\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:21.324714 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.322239    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f6130c-64f5-4dbc-9553-0cc11c35edd8-plugins-config-volume" pod "72f6130c-64f5-4dbc-9553-0cc11c35edd8" (UID: "72f6130c-64f5-4dbc-9553-0cc11c35edd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:21.328163 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.328134    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f6130c-64f5-4dbc-9553-0cc11c35edd8-kube-api-access-9w6f9" pod "72f6130c-64f5-4dbc-9553-0cc11c35edd8" (UID: "72f6130c-64f5-4dbc-9553-0cc11c35edd8"). InnerVolumeSpecName "kube-api-access-9w6f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:21.397542 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.397510    5510 generic.go:356] "Generic (PLEG): container finished" podID="72f6130c-64f5-4dbc-9553-0cc11c35edd8" containerID="ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6" exitCode=0
Nov 01 06:48:21.397704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.397562    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"
Nov 01 06:48:21.397704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.397626    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd" event={"ID":"72f6130c-64f5-4dbc-9553-0cc11c35edd8","Type":"ContainerDied","Data":"ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6"}
Nov 01 06:48:21.397704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.397687    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd" event={"ID":"72f6130c-64f5-4dbc-9553-0cc11c35edd8","Type":"ContainerDied","Data":"b198d6dd3bd2df5d7defc7c74994a9585298a2d6f6b3a451f7e31d888f711e94"}
Nov 01 06:48:21.397939 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.397717    5510 scope.go:122] "RemoveContainer" containerID="ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6"
Nov 01 06:48:21.399080 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.399056    5510 generic.go:356] "Generic (PLEG): container finished" podID="0bd3b495-c7e6-41e7-b06d-1014e71d0585" containerID="48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407" exitCode=0
Nov 01 06:48:21.399194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.399109    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"
Nov 01 06:48:21.399194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.399127    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg" event={"ID":"0bd3b495-c7e6-41e7-b06d-1014e71d0585","Type":"ContainerDied","Data":"48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407"}
Nov 01 06:48:21.399194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.399165    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg" event={"ID":"0bd3b495-c7e6-41e7-b06d-1014e71d0585","Type":"ContainerDied","Data":"cafd7600ddd26c963c57c9c0fb2342a3b0974a70b81b64fa54f2c728d2410fc7"}
Nov 01 06:48:21.407569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.407554    5510 scope.go:122] "RemoveContainer" containerID="ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6"
Nov 01 06:48:21.407822 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:48:21.407805    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6\": container with ID starting with ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6 not found: ID does not exist" containerID="ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6"
Nov 01 06:48:21.407893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.407843    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6"} err="failed to get container status \"ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6\": rpc error: code = NotFound desc = could not find container \"ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6\": container with ID starting with ebeb0820267d3b9b2c876d507cc37a9c1790007b22ef65dd56e31c045bd5e5b6 not found: ID does not exist"
Nov 01 06:48:21.407893 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.407859    5510 scope.go:122] "RemoveContainer" containerID="48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407"
Nov 01 06:48:21.416227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.416212    5510 scope.go:122] "RemoveContainer" containerID="48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407"
Nov 01 06:48:21.416462 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:48:21.416443    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407\": container with ID starting with 48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407 not found: ID does not exist" containerID="48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407"
Nov 01 06:48:21.416507 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.416471    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407"} err="failed to get container status \"48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407\": rpc error: code = NotFound desc = could not find container \"48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407\": container with ID starting with 48caf9c7b641d8b85721cd1d849f06ab18d688bd9186445bf3b5f055363b7407 not found: ID does not exist"
Nov 01 06:48:21.422104 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.422084    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w6f9\" (UniqueName: \"kubernetes.io/projected/72f6130c-64f5-4dbc-9553-0cc11c35edd8-kube-api-access-9w6f9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:21.422172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.422106    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6130c-64f5-4dbc-9553-0cc11c35edd8-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:21.560935 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.560902    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"]
Nov 01 06:48:21.563117 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.563095    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8f45a2-97cf84875-w7jmd"]
Nov 01 06:48:21.672017 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.671939    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"]
Nov 01 06:48:21.677064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:21.677039    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8f45a2-649ffb6b67-r2frg"]
Nov 01 06:48:23.235786 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:23.235755    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd3b495-c7e6-41e7-b06d-1014e71d0585" path="/var/lib/kubelet/pods/0bd3b495-c7e6-41e7-b06d-1014e71d0585/volumes"
Nov 01 06:48:23.236194 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:23.236180    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f6130c-64f5-4dbc-9553-0cc11c35edd8" path="/var/lib/kubelet/pods/72f6130c-64f5-4dbc-9553-0cc11c35edd8/volumes"
Nov 01 06:48:28.347630 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:48:28.347594    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:48:47.544275 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:48:47.544234    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:49:01.650199 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:01.650168    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"]
Nov 01 06:49:06.651059 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:06.651001    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s" podUID="59be7ba7-9f54-41d6-a50a-916a7605e821" containerName="payload-processing" containerID="cri-o://3a9f17eb47b6a886b5def3b27e6e6805cb4983bf4b578d1e18079afceed633c2" gracePeriod=25
Nov 01 06:49:08.311534 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:08.311498    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:49:10.181059 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.181018    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"]
Nov 01 06:49:10.181541 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.181275    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc" podUID="928d5d5f-9565-45d6-985a-c0bff8efa61b" containerName="maas-api" containerID="cri-o://de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57" gracePeriod=30
Nov 01 06:49:10.476610 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.476583    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:49:10.570427 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.570387    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928d5d5f-9565-45d6-985a-c0bff8efa61b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928d5d5f-9565-45d6-985a-c0bff8efa61b-openshift-service-ca\") pod \"928d5d5f-9565-45d6-985a-c0bff8efa61b\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") "
Nov 01 06:49:10.571460 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.571434    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-metrics-certs\") pod \"928d5d5f-9565-45d6-985a-c0bff8efa61b\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") "
Nov 01 06:49:10.575287 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.571331    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928d5d5f-9565-45d6-985a-c0bff8efa61b-openshift-service-ca" pod "928d5d5f-9565-45d6-985a-c0bff8efa61b" (UID: "928d5d5f-9565-45d6-985a-c0bff8efa61b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:49:10.575601 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.575582    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls\") pod \"928d5d5f-9565-45d6-985a-c0bff8efa61b\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") "
Nov 01 06:49:10.576105 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.576087    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928d5d5f-9565-45d6-985a-c0bff8efa61b-kube-api-access-tgqb7\" (UniqueName: \"kubernetes.io/projected/928d5d5f-9565-45d6-985a-c0bff8efa61b-kube-api-access-tgqb7\") pod \"928d5d5f-9565-45d6-985a-c0bff8efa61b\" (UID: \"928d5d5f-9565-45d6-985a-c0bff8efa61b\") "
Nov 01 06:49:10.576966 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.576947    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928d5d5f-9565-45d6-985a-c0bff8efa61b-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:10.582133 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.582089    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-metrics-certs" pod "928d5d5f-9565-45d6-985a-c0bff8efa61b" (UID: "928d5d5f-9565-45d6-985a-c0bff8efa61b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:49:10.584423 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.584387    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls" pod "928d5d5f-9565-45d6-985a-c0bff8efa61b" (UID: "928d5d5f-9565-45d6-985a-c0bff8efa61b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:49:10.604537 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.602686    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928d5d5f-9565-45d6-985a-c0bff8efa61b-kube-api-access-tgqb7" pod "928d5d5f-9565-45d6-985a-c0bff8efa61b" (UID: "928d5d5f-9565-45d6-985a-c0bff8efa61b"). InnerVolumeSpecName "kube-api-access-tgqb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:49:10.678177 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.678145    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:10.678177 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.678173    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgqb7\" (UniqueName: \"kubernetes.io/projected/928d5d5f-9565-45d6-985a-c0bff8efa61b-kube-api-access-tgqb7\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:10.678177 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.678185    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928d5d5f-9565-45d6-985a-c0bff8efa61b-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:10.740854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.740167    5510 generic.go:356] "Generic (PLEG): container finished" podID="928d5d5f-9565-45d6-985a-c0bff8efa61b" containerID="de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57" exitCode=0
Nov 01 06:49:10.740854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.740227    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"
Nov 01 06:49:10.740854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.740308    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc" event={"ID":"928d5d5f-9565-45d6-985a-c0bff8efa61b","Type":"ContainerDied","Data":"de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57"}
Nov 01 06:49:10.740854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.740352    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc" event={"ID":"928d5d5f-9565-45d6-985a-c0bff8efa61b","Type":"ContainerDied","Data":"63cbb3d9e9ad9ec5aa5d293b4fbdf92773252620ec3bf4898c0d67122e9437b2"}
Nov 01 06:49:10.740854 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.740378    5510 scope.go:122] "RemoveContainer" containerID="de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57"
Nov 01 06:49:10.754545 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.754516    5510 scope.go:122] "RemoveContainer" containerID="de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57"
Nov 01 06:49:10.754805 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:49:10.754782    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57\": container with ID starting with de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57 not found: ID does not exist" containerID="de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57"
Nov 01 06:49:10.754899 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.754814    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57"} err="failed to get container status \"de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57\": rpc error: code = NotFound desc = could not find container \"de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57\": container with ID starting with de5313fbdaed3e2b8b0fadf8a10590cb488d96e315d651088d956f694711bd57 not found: ID does not exist"
Nov 01 06:49:10.903981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.903943    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"]
Nov 01 06:49:10.909424 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:10.909394    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d25db-b5868b987-g8sgc"]
Nov 01 06:49:11.235716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:11.235687    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928d5d5f-9565-45d6-985a-c0bff8efa61b" path="/var/lib/kubelet/pods/928d5d5f-9565-45d6-985a-c0bff8efa61b/volumes"
Nov 01 06:49:13.709178 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:13.709091    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s" podUID="59be7ba7-9f54-41d6-a50a-916a7605e821" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:49:14.467422 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:14.467391    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"]
Nov 01 06:49:14.467600 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:14.467579    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy" containerID="cri-o://2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d" gracePeriod=30
Nov 01 06:49:17.544769 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:49:17.544729    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:49:19.723159 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.723132    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:49:19.794464 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794384    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-token\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-token\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.794464 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794433    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-envoy\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.794666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794490    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-podinfo\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.794666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794529    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istiod-ca-cert\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.794666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794587    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-credential-socket\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.794666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794630    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-data\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.794903 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794671    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-kube-api-access-798c5\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-kube-api-access-798c5\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.794903 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794713    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-socket\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.794903 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.794744    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-certs\") pod \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\" (UID: \"9e3b79fb-4201-4024-b2d4-fdeb139fef14\") "
Nov 01 06:49:19.795057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.795036    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-credential-socket" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:49:19.795336 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.795315    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:19.795442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.795418    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-certs" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:49:19.795968 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.795941    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-socket" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:49:19.796130 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.796106    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istiod-ca-cert" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:49:19.796662 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.796634    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-data" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:49:19.796662 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.796649    5510 generic.go:356] "Generic (PLEG): container finished" podID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerID="2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d" exitCode=0
Nov 01 06:49:19.796812 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.796745    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"
Nov 01 06:49:19.796914 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.796889    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" event={"ID":"9e3b79fb-4201-4024-b2d4-fdeb139fef14","Type":"ContainerDied","Data":"2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d"}
Nov 01 06:49:19.797026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.796949    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5" event={"ID":"9e3b79fb-4201-4024-b2d4-fdeb139fef14","Type":"ContainerDied","Data":"5b6664e44018d73d90c43e901a4e7ec08dd7ceb14cea82ccc1a0fa10eeebc171"}
Nov 01 06:49:19.797026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.796975    5510 scope.go:122] "RemoveContainer" containerID="2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d"
Nov 01 06:49:19.797155 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.797055    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-envoy" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:49:19.797453 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.797419    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-token" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:49:19.798451 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.798432    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-kube-api-access-798c5" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "kube-api-access-798c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:49:19.798926 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.798905    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-podinfo" pod "9e3b79fb-4201-4024-b2d4-fdeb139fef14" (UID: "9e3b79fb-4201-4024-b2d4-fdeb139fef14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:49:19.820865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.820837    5510 scope.go:122] "RemoveContainer" containerID="feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9"
Nov 01 06:49:19.829962 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.829945    5510 scope.go:122] "RemoveContainer" containerID="2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d"
Nov 01 06:49:19.830221 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:49:19.830196    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d\": container with ID starting with 2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d not found: ID does not exist" containerID="2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d"
Nov 01 06:49:19.830270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.830230    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d"} err="failed to get container status \"2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d\": rpc error: code = NotFound desc = could not find container \"2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d\": container with ID starting with 2ee6e9678310919ca89885f86f7f3f2a487f07b6fd203baf83e12ecca382905d not found: ID does not exist"
Nov 01 06:49:19.830270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.830247    5510 scope.go:122] "RemoveContainer" containerID="feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9"
Nov 01 06:49:19.830462 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:49:19.830439    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9\": container with ID starting with feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9 not found: ID does not exist" containerID="feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9"
Nov 01 06:49:19.830505 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.830465    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9"} err="failed to get container status \"feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9\": rpc error: code = NotFound desc = could not find container \"feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9\": container with ID starting with feb70427c95ea64d186d9be3342a17c395c49931e1daa18e1b63244da3d7e4b9 not found: ID does not exist"
Nov 01 06:49:19.895973 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.895945    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:19.895973 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.895969    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:19.896126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.895980    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:19.896126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.895995    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:19.896126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.896007    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:19.896126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.896017    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:19.896126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.896025    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-798c5\" (UniqueName: \"kubernetes.io/projected/9e3b79fb-4201-4024-b2d4-fdeb139fef14-kube-api-access-798c5\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:19.896126 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:19.896034    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3b79fb-4201-4024-b2d4-fdeb139fef14-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:20.216525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:20.216493    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"]
Nov 01 06:49:20.222346 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:20.222318    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d25db-openshift-default-5656468b5-25xv5"]
Nov 01 06:49:21.235442 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:21.235411    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" path="/var/lib/kubelet/pods/9e3b79fb-4201-4024-b2d4-fdeb139fef14/volumes"
Nov 01 06:49:23.708560 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:23.708522    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s" podUID="59be7ba7-9f54-41d6-a50a-916a7605e821" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:49:26.841287 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:26.841258    5510 generic.go:356] "Generic (PLEG): container finished" podID="59be7ba7-9f54-41d6-a50a-916a7605e821" containerID="3a9f17eb47b6a886b5def3b27e6e6805cb4983bf4b578d1e18079afceed633c2" exitCode=0
Nov 01 06:49:26.841676 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:26.841298    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s" event={"ID":"59be7ba7-9f54-41d6-a50a-916a7605e821","Type":"ContainerDied","Data":"3a9f17eb47b6a886b5def3b27e6e6805cb4983bf4b578d1e18079afceed633c2"}
Nov 01 06:49:26.906137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:26.906115    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:49:27.074071 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.074042    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59be7ba7-9f54-41d6-a50a-916a7605e821-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59be7ba7-9f54-41d6-a50a-916a7605e821-plugins-config-volume\") pod \"59be7ba7-9f54-41d6-a50a-916a7605e821\" (UID: \"59be7ba7-9f54-41d6-a50a-916a7605e821\") "
Nov 01 06:49:27.074265 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.074192    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59be7ba7-9f54-41d6-a50a-916a7605e821-kube-api-access-l7qfp\" (UniqueName: \"kubernetes.io/projected/59be7ba7-9f54-41d6-a50a-916a7605e821-kube-api-access-l7qfp\") pod \"59be7ba7-9f54-41d6-a50a-916a7605e821\" (UID: \"59be7ba7-9f54-41d6-a50a-916a7605e821\") "
Nov 01 06:49:27.074412 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.074390    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59be7ba7-9f54-41d6-a50a-916a7605e821-plugins-config-volume" pod "59be7ba7-9f54-41d6-a50a-916a7605e821" (UID: "59be7ba7-9f54-41d6-a50a-916a7605e821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:49:27.074506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.074494    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59be7ba7-9f54-41d6-a50a-916a7605e821-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:27.076418 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.076395    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59be7ba7-9f54-41d6-a50a-916a7605e821-kube-api-access-l7qfp" pod "59be7ba7-9f54-41d6-a50a-916a7605e821" (UID: "59be7ba7-9f54-41d6-a50a-916a7605e821"). InnerVolumeSpecName "kube-api-access-l7qfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:49:27.175408 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.175372    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qfp\" (UniqueName: \"kubernetes.io/projected/59be7ba7-9f54-41d6-a50a-916a7605e821-kube-api-access-l7qfp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:27.849197 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.849169    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"
Nov 01 06:49:27.849651 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.849195    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s" event={"ID":"59be7ba7-9f54-41d6-a50a-916a7605e821","Type":"ContainerDied","Data":"a454a426e4c84d0e9faad253fc9ea6e65c38e7bbd14f668c798c8703a3d857e2"}
Nov 01 06:49:27.849651 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:27.849255    5510 scope.go:122] "RemoveContainer" containerID="3a9f17eb47b6a886b5def3b27e6e6805cb4983bf4b578d1e18079afceed633c2"
Nov 01 06:49:28.018488 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:28.018426    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"]
Nov 01 06:49:28.022438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:28.022410    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d25db-665dbd774f-hz49s"]
Nov 01 06:49:29.235380 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:29.235348    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59be7ba7-9f54-41d6-a50a-916a7605e821" path="/var/lib/kubelet/pods/59be7ba7-9f54-41d6-a50a-916a7605e821/volumes"
Nov 01 06:49:29.289046 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:29.289017    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"]
Nov 01 06:49:29.392666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:29.392636    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"]
Nov 01 06:49:34.102636 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.102590    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"]
Nov 01 06:49:34.103275 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103258    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8d987f61-f6ba-4950-9ba2-7023096db483" containerName="payload-processing"
Nov 01 06:49:34.103348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103277    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d987f61-f6ba-4950-9ba2-7023096db483" containerName="payload-processing"
Nov 01 06:49:34.103348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103295    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5e7eaa36-4938-4d11-8db2-304a18f0de38" containerName="payload-pre-processing"
Nov 01 06:49:34.103348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103301    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7eaa36-4938-4d11-8db2-304a18f0de38" containerName="payload-pre-processing"
Nov 01 06:49:34.103348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103319    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy"
Nov 01 06:49:34.103348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103324    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy"
Nov 01 06:49:34.103348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103333    5510 cpu_manager.go:414] "containerMap: removing container" podUID="59be7ba7-9f54-41d6-a50a-916a7605e821" containerName="payload-processing"
Nov 01 06:49:34.103348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103338    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59be7ba7-9f54-41d6-a50a-916a7605e821" containerName="payload-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103353    5510 cpu_manager.go:414] "containerMap: removing container" podUID="928d5d5f-9565-45d6-985a-c0bff8efa61b" containerName="maas-api"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103358    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928d5d5f-9565-45d6-985a-c0bff8efa61b" containerName="maas-api"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103365    5510 cpu_manager.go:414] "containerMap: removing container" podUID="72f6130c-64f5-4dbc-9553-0cc11c35edd8" containerName="payload-pre-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103371    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f6130c-64f5-4dbc-9553-0cc11c35edd8" containerName="payload-pre-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103381    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0bd3b495-c7e6-41e7-b06d-1014e71d0585" containerName="payload-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103386    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd3b495-c7e6-41e7-b06d-1014e71d0585" containerName="payload-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103458    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103467    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="59be7ba7-9f54-41d6-a50a-916a7605e821" containerName="payload-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103475    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="928d5d5f-9565-45d6-985a-c0bff8efa61b" containerName="maas-api"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103480    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f6130c-64f5-4dbc-9553-0cc11c35edd8" containerName="payload-pre-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103486    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103491    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd3b495-c7e6-41e7-b06d-1014e71d0585" containerName="payload-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103497    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d987f61-f6ba-4950-9ba2-7023096db483" containerName="payload-processing"
Nov 01 06:49:34.103562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.103504    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7eaa36-4938-4d11-8db2-304a18f0de38" containerName="payload-pre-processing"
Nov 01 06:49:34.108260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.108245    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.162739 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.162710    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2260d\"/\"openshift-service-ca.crt\""
Nov 01 06:49:34.162930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.162710    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2260d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d2260d\""
Nov 01 06:49:34.212500 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.212452    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2260d\"/\"routing-overlay\""
Nov 01 06:49:34.212644 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.212543    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2260d\"/\"kube-root-ca.crt\""
Nov 01 06:49:34.212644 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.212548    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2260d\"/\"agc-cred-bf0963edd435\""
Nov 01 06:49:34.212644 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.212601    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2260d\"/\"payload-processing-external-model-e2e--637b3180-dockercfg-6tqhp\""
Nov 01 06:49:34.214765 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.214744    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"]
Nov 01 06:49:34.273305 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.273278    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.273445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.273322    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqzd6\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-kube-api-access-lqzd6\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.273445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.273438    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.273575 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.273518    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.289510 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.289446    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q" podUID="e7176c24-60a4-4250-822f-84fd0b79b881" containerName="payload-processing" containerID="cri-o://b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3" gracePeriod=25
Nov 01 06:49:34.374462 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.374399    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.374462 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.374436    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqzd6\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-kube-api-access-lqzd6\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.374462 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.374463    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.374717 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.374506    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.375195 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.375173    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.375348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.375283    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.376924 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.376907    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.393400 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.393356    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq" podUID="a82ed2f5-f019-4457-a3f9-b99da85c104f" containerName="payload-pre-processing" containerID="cri-o://a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77" gracePeriod=25
Nov 01 06:49:34.434977 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.434950    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqzd6\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-kube-api-access-lqzd6\") pod \"payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") " pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:34.722440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:34.722417    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:35.097632 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:35.097607    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"]
Nov 01 06:49:35.100731 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:49:35.100703    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfce2fb8_cb98_490e_8b04_6e2a45e2910e.slice/crio-712e742804a6bceaeb8c937f9247408beb7e5af3bd12a97ec059fb56a643d9b4 WatchSource:0}: Error finding container 712e742804a6bceaeb8c937f9247408beb7e5af3bd12a97ec059fb56a643d9b4: Status 404 returned error can't find the container with id 712e742804a6bceaeb8c937f9247408beb7e5af3bd12a97ec059fb56a643d9b4
Nov 01 06:49:35.919907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:35.919866    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w" event={"ID":"cfce2fb8-cb98-490e-8b04-6e2a45e2910e","Type":"ContainerStarted","Data":"50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5"}
Nov 01 06:49:35.920345 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:35.919925    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w" event={"ID":"cfce2fb8-cb98-490e-8b04-6e2a45e2910e","Type":"ContainerStarted","Data":"712e742804a6bceaeb8c937f9247408beb7e5af3bd12a97ec059fb56a643d9b4"}
Nov 01 06:49:35.920345 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:35.919946    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:36.037452 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:36.037390    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w" podStartSLOduration=2.037376516 podStartE2EDuration="2.037376516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:49:36.035440533 +0000 UTC m=+3812.913413939" watchObservedRunningTime="2026-11-01 06:49:36.037376516 +0000 UTC m=+3812.915349926"
Nov 01 06:49:40.453350 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:40.453316    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:49:41.333446 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:41.329295    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q" podUID="e7176c24-60a4-4250-822f-84fd0b79b881" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:49:41.336678 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:41.336645    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq" podUID="a82ed2f5-f019-4457-a3f9-b99da85c104f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:49:46.928877 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:46.928847    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:49:47.544818 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:49:47.544783    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:49:51.323924 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:51.323884    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q" podUID="e7176c24-60a4-4250-822f-84fd0b79b881" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:49:51.323924 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:51.323898    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq" podUID="a82ed2f5-f019-4457-a3f9-b99da85c104f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:49:54.568048 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.568022    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 06:49:54.664740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.664717    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 06:49:54.706929 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.706901    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7176c24-60a4-4250-822f-84fd0b79b881-kube-api-access-t95fc\" (UniqueName: \"kubernetes.io/projected/e7176c24-60a4-4250-822f-84fd0b79b881-kube-api-access-t95fc\") pod \"e7176c24-60a4-4250-822f-84fd0b79b881\" (UID: \"e7176c24-60a4-4250-822f-84fd0b79b881\") "
Nov 01 06:49:54.707089 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.707024    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7176c24-60a4-4250-822f-84fd0b79b881-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7176c24-60a4-4250-822f-84fd0b79b881-plugins-config-volume\") pod \"e7176c24-60a4-4250-822f-84fd0b79b881\" (UID: \"e7176c24-60a4-4250-822f-84fd0b79b881\") "
Nov 01 06:49:54.707398 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.707372    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7176c24-60a4-4250-822f-84fd0b79b881-plugins-config-volume" pod "e7176c24-60a4-4250-822f-84fd0b79b881" (UID: "e7176c24-60a4-4250-822f-84fd0b79b881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:49:54.709201 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.709180    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7176c24-60a4-4250-822f-84fd0b79b881-kube-api-access-t95fc" pod "e7176c24-60a4-4250-822f-84fd0b79b881" (UID: "e7176c24-60a4-4250-822f-84fd0b79b881"). InnerVolumeSpecName "kube-api-access-t95fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:49:54.808189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.808122    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82ed2f5-f019-4457-a3f9-b99da85c104f-kube-api-access-hbp8g\" (UniqueName: \"kubernetes.io/projected/a82ed2f5-f019-4457-a3f9-b99da85c104f-kube-api-access-hbp8g\") pod \"a82ed2f5-f019-4457-a3f9-b99da85c104f\" (UID: \"a82ed2f5-f019-4457-a3f9-b99da85c104f\") "
Nov 01 06:49:54.808189 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.808170    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82ed2f5-f019-4457-a3f9-b99da85c104f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ed2f5-f019-4457-a3f9-b99da85c104f-plugins-config-volume\") pod \"a82ed2f5-f019-4457-a3f9-b99da85c104f\" (UID: \"a82ed2f5-f019-4457-a3f9-b99da85c104f\") "
Nov 01 06:49:54.808613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.808589    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82ed2f5-f019-4457-a3f9-b99da85c104f-plugins-config-volume" pod "a82ed2f5-f019-4457-a3f9-b99da85c104f" (UID: "a82ed2f5-f019-4457-a3f9-b99da85c104f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:49:54.808613 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.808604    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t95fc\" (UniqueName: \"kubernetes.io/projected/e7176c24-60a4-4250-822f-84fd0b79b881-kube-api-access-t95fc\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:54.808710 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.808630    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7176c24-60a4-4250-822f-84fd0b79b881-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:54.810542 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.810517    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82ed2f5-f019-4457-a3f9-b99da85c104f-kube-api-access-hbp8g" pod "a82ed2f5-f019-4457-a3f9-b99da85c104f" (UID: "a82ed2f5-f019-4457-a3f9-b99da85c104f"). InnerVolumeSpecName "kube-api-access-hbp8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:49:54.910207 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.910177    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbp8g\" (UniqueName: \"kubernetes.io/projected/a82ed2f5-f019-4457-a3f9-b99da85c104f-kube-api-access-hbp8g\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:54.910207 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:54.910203    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ed2f5-f019-4457-a3f9-b99da85c104f-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:55.039622 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.039590    5510 generic.go:356] "Generic (PLEG): container finished" podID="e7176c24-60a4-4250-822f-84fd0b79b881" containerID="b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3" exitCode=0
Nov 01 06:49:55.039752 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.039657    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"
Nov 01 06:49:55.039752 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.039673    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q" event={"ID":"e7176c24-60a4-4250-822f-84fd0b79b881","Type":"ContainerDied","Data":"b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3"}
Nov 01 06:49:55.039752 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.039741    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q" event={"ID":"e7176c24-60a4-4250-822f-84fd0b79b881","Type":"ContainerDied","Data":"ed7d6a84d1d1d8842fd473ffe32da59b0e7ee062dc03d932868a95e01a9aac4e"}
Nov 01 06:49:55.039928 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.039762    5510 scope.go:122] "RemoveContainer" containerID="b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3"
Nov 01 06:49:55.041166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.041143    5510 generic.go:356] "Generic (PLEG): container finished" podID="a82ed2f5-f019-4457-a3f9-b99da85c104f" containerID="a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77" exitCode=0
Nov 01 06:49:55.041279 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.041206    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq" event={"ID":"a82ed2f5-f019-4457-a3f9-b99da85c104f","Type":"ContainerDied","Data":"a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77"}
Nov 01 06:49:55.041279 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.041243    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq" event={"ID":"a82ed2f5-f019-4457-a3f9-b99da85c104f","Type":"ContainerDied","Data":"dc45f9d23cb32b99391e4d6b794d737fd83bd722e2829340199c8abfc2a8c515"}
Nov 01 06:49:55.041387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.041368    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"
Nov 01 06:49:55.050273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.050251    5510 scope.go:122] "RemoveContainer" containerID="b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3"
Nov 01 06:49:55.050574 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:49:55.050548    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3\": container with ID starting with b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3 not found: ID does not exist" containerID="b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3"
Nov 01 06:49:55.050649 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.050583    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3"} err="failed to get container status \"b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3\": rpc error: code = NotFound desc = could not find container \"b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3\": container with ID starting with b13714cf89064a0a585ed8df31997d45111244727463fae8939cb0a67d6af3e3 not found: ID does not exist"
Nov 01 06:49:55.050649 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.050601    5510 scope.go:122] "RemoveContainer" containerID="a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77"
Nov 01 06:49:55.059859 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.059817    5510 scope.go:122] "RemoveContainer" containerID="a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77"
Nov 01 06:49:55.060109 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:49:55.060092    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77\": container with ID starting with a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77 not found: ID does not exist" containerID="a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77"
Nov 01 06:49:55.060169 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.060116    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77"} err="failed to get container status \"a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77\": rpc error: code = NotFound desc = could not find container \"a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77\": container with ID starting with a16c7d1781824a6f61714afedf2d4988f1819088dc0230ec299fbbb6a134fd77 not found: ID does not exist"
Nov 01 06:49:55.202561 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.202528    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"]
Nov 01 06:49:55.252010 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.251977    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-14a6e4-7bdff86d5f-87gjq"]
Nov 01 06:49:55.314212 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.314148    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"]
Nov 01 06:49:55.318155 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:55.318135    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-14a6e4-5854b955fc-dr86q"]
Nov 01 06:49:57.236049 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:57.236019    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82ed2f5-f019-4457-a3f9-b99da85c104f" path="/var/lib/kubelet/pods/a82ed2f5-f019-4457-a3f9-b99da85c104f/volumes"
Nov 01 06:49:57.236436 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:49:57.236410    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7176c24-60a4-4250-822f-84fd0b79b881" path="/var/lib/kubelet/pods/e7176c24-60a4-4250-822f-84fd0b79b881/volumes"
Nov 01 06:50:05.645571 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:05.645534    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:50:06.501185 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:06.501152    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"]
Nov 01 06:50:06.501422 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:06.501376    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy" containerID="cri-o://5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a" gracePeriod=30
Nov 01 06:50:09.138633 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:09.138597    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"]
Nov 01 06:50:09.139071 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:09.138807    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy" containerID="cri-o://798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7" gracePeriod=30
Nov 01 06:50:11.777573 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.777512    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 06:50:11.814971 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.814938    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-istio-token\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-istio-token\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815024    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-certs\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815061    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-kube-api-access-mch2q\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-kube-api-access-mch2q\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815083    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-socket\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815103    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-envoy\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815166 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815125    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-credential-socket\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815168    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75de03a3-8add-411c-a293-9487c9b218ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75de03a3-8add-411c-a293-9487c9b218ad-istio-podinfo\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815215    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75de03a3-8add-411c-a293-9487c9b218ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75de03a3-8add-411c-a293-9487c9b218ad-istiod-ca-cert\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815238    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-data\") pod \"75de03a3-8add-411c-a293-9487c9b218ad\" (UID: \"75de03a3-8add-411c-a293-9487c9b218ad\") "
Nov 01 06:50:11.815587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815431    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-certs" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.815587 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815452    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-socket" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.815776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815725    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.815776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815749    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.815776 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815746    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-credential-socket" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.815978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815821    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-data" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.815978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.815954    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75de03a3-8add-411c-a293-9487c9b218ad-istiod-ca-cert" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:11.817779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.817752    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75de03a3-8add-411c-a293-9487c9b218ad-istio-podinfo" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:50:11.817779 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.817764    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-istio-token" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:11.818057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.818034    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-kube-api-access-mch2q" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "kube-api-access-mch2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:11.818107 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.818048    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-envoy" pod "75de03a3-8add-411c-a293-9487c9b218ad" (UID: "75de03a3-8add-411c-a293-9487c9b218ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.917387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.917351    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.917387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.917380    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mch2q\" (UniqueName: \"kubernetes.io/projected/75de03a3-8add-411c-a293-9487c9b218ad-kube-api-access-mch2q\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.917387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.917391    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.917650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.917400    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.917650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.917409    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75de03a3-8add-411c-a293-9487c9b218ad-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.917650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.917417    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75de03a3-8add-411c-a293-9487c9b218ad-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.917650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:11.917425    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75de03a3-8add-411c-a293-9487c9b218ad-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:12.176217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.176127    5510 generic.go:356] "Generic (PLEG): container finished" podID="75de03a3-8add-411c-a293-9487c9b218ad" containerID="5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a" exitCode=0
Nov 01 06:50:12.176217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.176174    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"
Nov 01 06:50:12.176217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.176209    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" event={"ID":"75de03a3-8add-411c-a293-9487c9b218ad","Type":"ContainerDied","Data":"5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a"}
Nov 01 06:50:12.176444 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.176265    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj" event={"ID":"75de03a3-8add-411c-a293-9487c9b218ad","Type":"ContainerDied","Data":"da90a739dd842f82792e369116924ecbe024f405a64b871acd6df3dae99d78c3"}
Nov 01 06:50:12.176444 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.176295    5510 scope.go:122] "RemoveContainer" containerID="5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a"
Nov 01 06:50:12.187264 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.187245    5510 scope.go:122] "RemoveContainer" containerID="a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac"
Nov 01 06:50:12.198394 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.198371    5510 scope.go:122] "RemoveContainer" containerID="5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a"
Nov 01 06:50:12.198686 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:50:12.198661    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a\": container with ID starting with 5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a not found: ID does not exist" containerID="5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a"
Nov 01 06:50:12.198780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.198696    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a"} err="failed to get container status \"5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a\": rpc error: code = NotFound desc = could not find container \"5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a\": container with ID starting with 5f2774b273b8983e62b7feeae87002ead8c3ea6882a95af3e84013483e41ab7a not found: ID does not exist"
Nov 01 06:50:12.198780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.198722    5510 scope.go:122] "RemoveContainer" containerID="a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac"
Nov 01 06:50:12.199037 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:50:12.199014    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac\": container with ID starting with a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac not found: ID does not exist" containerID="a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac"
Nov 01 06:50:12.199084 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.199045    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac"} err="failed to get container status \"a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac\": rpc error: code = NotFound desc = could not find container \"a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac\": container with ID starting with a3a0468fade1ca6303c991359c856bce75113108582034ee5474eeb0f5d8a6ac not found: ID does not exist"
Nov 01 06:50:12.339452 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.339414    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"]
Nov 01 06:50:12.345734 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:12.345704    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-2pbtj"]
Nov 01 06:50:13.235301 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:13.235269    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75de03a3-8add-411c-a293-9487c9b218ad" path="/var/lib/kubelet/pods/75de03a3-8add-411c-a293-9487c9b218ad/volumes"
Nov 01 06:50:14.387270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.387246    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:50:14.443489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443413    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-kube-api-access-dkpb7\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-kube-api-access-dkpb7\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.443489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443478    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-credential-socket\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.443722 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443525    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-socket\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.443722 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443589    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-data\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.443722 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443651    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-podinfo\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.443722 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443691    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-envoy\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.444007 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443725    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-certs\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.444007 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443768    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-token\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-token\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.444007 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443814    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-credential-socket" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.444007 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443869    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b571d511-15fb-4937-bc74-2c808ab8fec3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b571d511-15fb-4937-bc74-2c808ab8fec3-istiod-ca-cert\") pod \"b571d511-15fb-4937-bc74-2c808ab8fec3\" (UID: \"b571d511-15fb-4937-bc74-2c808ab8fec3\") "
Nov 01 06:50:14.444235 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.443869    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-socket" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.444235 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.444143    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-data" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.444235 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.444062    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-certs" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.444410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.444317    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.444410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.444341    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.444410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.444361    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.444410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.444374    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.444410 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.444385    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b571d511-15fb-4937-bc74-2c808ab8fec3-istiod-ca-cert" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:14.445946 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.445919    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-kube-api-access-dkpb7" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "kube-api-access-dkpb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:14.446199 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.446183    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-podinfo" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:50:14.446693 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.446672    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-token" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:14.446791 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.446670    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-envoy" pod "b571d511-15fb-4937-bc74-2c808ab8fec3" (UID: "b571d511-15fb-4937-bc74-2c808ab8fec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.545223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.545191    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.545223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.545219    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.545223 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.545229    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.545472 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.545240    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b571d511-15fb-4937-bc74-2c808ab8fec3-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.545472 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:14.545249    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkpb7\" (UniqueName: \"kubernetes.io/projected/b571d511-15fb-4937-bc74-2c808ab8fec3-kube-api-access-dkpb7\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:15.195788 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.195746    5510 generic.go:356] "Generic (PLEG): container finished" podID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerID="798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7" exitCode=0
Nov 01 06:50:15.195979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.195817    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"
Nov 01 06:50:15.195979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.195845    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" event={"ID":"b571d511-15fb-4937-bc74-2c808ab8fec3","Type":"ContainerDied","Data":"798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7"}
Nov 01 06:50:15.195979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.195934    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h" event={"ID":"b571d511-15fb-4937-bc74-2c808ab8fec3","Type":"ContainerDied","Data":"4200d1a030a706533df294dbee4f290739cf6b17ee7af232d152f6b9fcc92ab5"}
Nov 01 06:50:15.195979 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.195957    5510 scope.go:122] "RemoveContainer" containerID="798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7"
Nov 01 06:50:15.206240 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.206221    5510 scope.go:122] "RemoveContainer" containerID="798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7"
Nov 01 06:50:15.206537 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:50:15.206516    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7\": container with ID starting with 798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7 not found: ID does not exist" containerID="798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7"
Nov 01 06:50:15.206634 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.206543    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7"} err="failed to get container status \"798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7\": rpc error: code = NotFound desc = could not find container \"798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7\": container with ID starting with 798fe252771446c4a5aac8bd3651ddb69d79db692bd4323fc7e97b5741f609c7 not found: ID does not exist"
Nov 01 06:50:15.359353 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.359322    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"]
Nov 01 06:50:15.408899 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:15.408861    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-2t54h"]
Nov 01 06:50:17.236626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:17.236597    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" path="/var/lib/kubelet/pods/b571d511-15fb-4937-bc74-2c808ab8fec3/volumes"
Nov 01 06:50:17.545624 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:50:17.545526    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:50:24.743986 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:24.743955    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:50:41.586578 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:41.586544    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"]
Nov 01 06:50:46.587916 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.587812    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w" podUID="cfce2fb8-cb98-490e-8b04-6e2a45e2910e" containerName="payload-processing" containerID="cri-o://50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5" gracePeriod=25
Nov 01 06:50:46.853660 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.853599    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:50:46.888356 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.888322    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-plugins-config-volume\") pod \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") "
Nov 01 06:50:46.888530 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.888412    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-provider-credentials\") pod \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") "
Nov 01 06:50:46.888530 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.888467    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-routing-overlay-volume\") pod \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") "
Nov 01 06:50:46.888656 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.888623    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-kube-api-access-lqzd6\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-kube-api-access-lqzd6\") pod \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\" (UID: \"cfce2fb8-cb98-490e-8b04-6e2a45e2910e\") "
Nov 01 06:50:46.888808 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.888781    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-plugins-config-volume" pod "cfce2fb8-cb98-490e-8b04-6e2a45e2910e" (UID: "cfce2fb8-cb98-490e-8b04-6e2a45e2910e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:46.888908 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.888795    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-routing-overlay-volume" pod "cfce2fb8-cb98-490e-8b04-6e2a45e2910e" (UID: "cfce2fb8-cb98-490e-8b04-6e2a45e2910e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:46.889050 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.889033    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:46.889096 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.889057    5510 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-routing-overlay-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:46.890995 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.890970    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-kube-api-access-lqzd6" pod "cfce2fb8-cb98-490e-8b04-6e2a45e2910e" (UID: "cfce2fb8-cb98-490e-8b04-6e2a45e2910e"). InnerVolumeSpecName "kube-api-access-lqzd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:46.891155 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.891132    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-provider-credentials" pod "cfce2fb8-cb98-490e-8b04-6e2a45e2910e" (UID: "cfce2fb8-cb98-490e-8b04-6e2a45e2910e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:46.990413 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.990383    5510 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-provider-credentials\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:46.990413 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:46.990409    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqzd6\" (UniqueName: \"kubernetes.io/projected/cfce2fb8-cb98-490e-8b04-6e2a45e2910e-kube-api-access-lqzd6\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:47.371642 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.371608    5510 generic.go:356] "Generic (PLEG): container finished" podID="cfce2fb8-cb98-490e-8b04-6e2a45e2910e" containerID="50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5" exitCode=0
Nov 01 06:50:47.371813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.371663    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"
Nov 01 06:50:47.371813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.371692    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w" event={"ID":"cfce2fb8-cb98-490e-8b04-6e2a45e2910e","Type":"ContainerDied","Data":"50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5"}
Nov 01 06:50:47.371813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.371746    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w" event={"ID":"cfce2fb8-cb98-490e-8b04-6e2a45e2910e","Type":"ContainerDied","Data":"712e742804a6bceaeb8c937f9247408beb7e5af3bd12a97ec059fb56a643d9b4"}
Nov 01 06:50:47.371813 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.371769    5510 scope.go:122] "RemoveContainer" containerID="50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5"
Nov 01 06:50:47.380770 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.380753    5510 scope.go:122] "RemoveContainer" containerID="50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5"
Nov 01 06:50:47.381071 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:50:47.381052    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5\": container with ID starting with 50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5 not found: ID does not exist" containerID="50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5"
Nov 01 06:50:47.381140 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.381078    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5"} err="failed to get container status \"50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5\": rpc error: code = NotFound desc = could not find container \"50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5\": container with ID starting with 50e94eab6c877eaaecde3692b365e5a3febd6c9679726ceb36da2f21f810e2b5 not found: ID does not exist"
Nov 01 06:50:47.535707 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.535612    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"]
Nov 01 06:50:47.545819 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:50:47.545784    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:50:47.584839 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:47.584795    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2260d/payload-processing-external-model-e2e-extmod-b-d2260d-6ff5mrp8w"]
Nov 01 06:50:48.840935 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:48.840903    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:50:49.236229 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:50:49.236200    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfce2fb8-cb98-490e-8b04-6e2a45e2910e" path="/var/lib/kubelet/pods/cfce2fb8-cb98-490e-8b04-6e2a45e2910e/volumes"
Nov 01 06:51:12.568955 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:12.568921    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:51:17.545946 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:51:17.545905    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:51:35.916728 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:35.916691    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:51:37.287689 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:37.287654    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"]
Nov 01 06:51:37.392050 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:37.392016    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"]
Nov 01 06:51:42.288232 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:42.288164    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8" podUID="f179c728-3846-4be0-b477-6dc3d9b7ec46" containerName="payload-processing" containerID="cri-o://f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2" gracePeriod=25
Nov 01 06:51:42.392510 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:42.392471    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4" podUID="6738a9a2-4627-4e04-af0a-ddb34fc205be" containerName="payload-pre-processing" containerID="cri-o://e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088" gracePeriod=25
Nov 01 06:51:43.646719 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:43.646680    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8" podUID="f179c728-3846-4be0-b477-6dc3d9b7ec46" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:51:43.647154 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:43.646695    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4" podUID="6738a9a2-4627-4e04-af0a-ddb34fc205be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:51:47.566596 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:51:47.566554    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:51:53.645911 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:53.645867    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4" podUID="6738a9a2-4627-4e04-af0a-ddb34fc205be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:51:53.645911 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:53.645899    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8" podUID="f179c728-3846-4be0-b477-6dc3d9b7ec46" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:51:55.785465 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:51:55.785426    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:52:02.554915 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.554855    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:52:02.619271 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.619239    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f179c728-3846-4be0-b477-6dc3d9b7ec46-kube-api-access-wgx92\" (UniqueName: \"kubernetes.io/projected/f179c728-3846-4be0-b477-6dc3d9b7ec46-kube-api-access-wgx92\") pod \"f179c728-3846-4be0-b477-6dc3d9b7ec46\" (UID: \"f179c728-3846-4be0-b477-6dc3d9b7ec46\") "
Nov 01 06:52:02.619441 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.619306    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f179c728-3846-4be0-b477-6dc3d9b7ec46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f179c728-3846-4be0-b477-6dc3d9b7ec46-plugins-config-volume\") pod \"f179c728-3846-4be0-b477-6dc3d9b7ec46\" (UID: \"f179c728-3846-4be0-b477-6dc3d9b7ec46\") "
Nov 01 06:52:02.619725 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.619703    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f179c728-3846-4be0-b477-6dc3d9b7ec46-plugins-config-volume" pod "f179c728-3846-4be0-b477-6dc3d9b7ec46" (UID: "f179c728-3846-4be0-b477-6dc3d9b7ec46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:02.621559 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.621532    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f179c728-3846-4be0-b477-6dc3d9b7ec46-kube-api-access-wgx92" pod "f179c728-3846-4be0-b477-6dc3d9b7ec46" (UID: "f179c728-3846-4be0-b477-6dc3d9b7ec46"). InnerVolumeSpecName "kube-api-access-wgx92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:02.637702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.637683    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:52:02.720262 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.720230    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6738a9a2-4627-4e04-af0a-ddb34fc205be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a9a2-4627-4e04-af0a-ddb34fc205be-plugins-config-volume\") pod \"6738a9a2-4627-4e04-af0a-ddb34fc205be\" (UID: \"6738a9a2-4627-4e04-af0a-ddb34fc205be\") "
Nov 01 06:52:02.720440 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.720348    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6738a9a2-4627-4e04-af0a-ddb34fc205be-kube-api-access-8fm5k\" (UniqueName: \"kubernetes.io/projected/6738a9a2-4627-4e04-af0a-ddb34fc205be-kube-api-access-8fm5k\") pod \"6738a9a2-4627-4e04-af0a-ddb34fc205be\" (UID: \"6738a9a2-4627-4e04-af0a-ddb34fc205be\") "
Nov 01 06:52:02.720614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.720584    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f179c728-3846-4be0-b477-6dc3d9b7ec46-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:02.720614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.720599    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgx92\" (UniqueName: \"kubernetes.io/projected/f179c728-3846-4be0-b477-6dc3d9b7ec46-kube-api-access-wgx92\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:02.720729 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.720640    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6738a9a2-4627-4e04-af0a-ddb34fc205be-plugins-config-volume" pod "6738a9a2-4627-4e04-af0a-ddb34fc205be" (UID: "6738a9a2-4627-4e04-af0a-ddb34fc205be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:02.722736 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.722711    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6738a9a2-4627-4e04-af0a-ddb34fc205be-kube-api-access-8fm5k" pod "6738a9a2-4627-4e04-af0a-ddb34fc205be" (UID: "6738a9a2-4627-4e04-af0a-ddb34fc205be"). InnerVolumeSpecName "kube-api-access-8fm5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:02.779540 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.779509    5510 generic.go:356] "Generic (PLEG): container finished" podID="6738a9a2-4627-4e04-af0a-ddb34fc205be" containerID="e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088" exitCode=0
Nov 01 06:52:02.779665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.779550    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"
Nov 01 06:52:02.779665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.779591    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4" event={"ID":"6738a9a2-4627-4e04-af0a-ddb34fc205be","Type":"ContainerDied","Data":"e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088"}
Nov 01 06:52:02.779665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.779641    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4" event={"ID":"6738a9a2-4627-4e04-af0a-ddb34fc205be","Type":"ContainerDied","Data":"598b9fe524bf91d859e5f155c351dc2dd88792104b7f38266af61f31477656b8"}
Nov 01 06:52:02.779665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.779663    5510 scope.go:122] "RemoveContainer" containerID="e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088"
Nov 01 06:52:02.781063 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.781039    5510 generic.go:356] "Generic (PLEG): container finished" podID="f179c728-3846-4be0-b477-6dc3d9b7ec46" containerID="f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2" exitCode=0
Nov 01 06:52:02.781167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.781064    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8" event={"ID":"f179c728-3846-4be0-b477-6dc3d9b7ec46","Type":"ContainerDied","Data":"f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2"}
Nov 01 06:52:02.781167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.781097    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"
Nov 01 06:52:02.781167 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.781104    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8" event={"ID":"f179c728-3846-4be0-b477-6dc3d9b7ec46","Type":"ContainerDied","Data":"25715929f2f4342b4603dfa9cb4418491ef47a483148316c8b188da0b4c23c8b"}
Nov 01 06:52:02.793214 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.793196    5510 scope.go:122] "RemoveContainer" containerID="e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088"
Nov 01 06:52:02.793769 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:52:02.793744    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088\": container with ID starting with e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088 not found: ID does not exist" containerID="e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088"
Nov 01 06:52:02.793877 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.793777    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088"} err="failed to get container status \"e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088\": rpc error: code = NotFound desc = could not find container \"e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088\": container with ID starting with e95f61715f417d174a6a874947d4d6d2e9ba8bba18c5053c4d85f0e4c5757088 not found: ID does not exist"
Nov 01 06:52:02.793877 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.793800    5510 scope.go:122] "RemoveContainer" containerID="f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2"
Nov 01 06:52:02.802957 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.802936    5510 scope.go:122] "RemoveContainer" containerID="f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2"
Nov 01 06:52:02.803244 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:52:02.803220    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2\": container with ID starting with f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2 not found: ID does not exist" containerID="f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2"
Nov 01 06:52:02.803311 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.803254    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2"} err="failed to get container status \"f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2\": rpc error: code = NotFound desc = could not find container \"f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2\": container with ID starting with f7a84274126ada9b3e6d76ea80238c0c76fba75f9c77a7c1e19b99305e17b8a2 not found: ID does not exist"
Nov 01 06:52:02.821270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.821217    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fm5k\" (UniqueName: \"kubernetes.io/projected/6738a9a2-4627-4e04-af0a-ddb34fc205be-kube-api-access-8fm5k\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:02.821270 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.821242    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a9a2-4627-4e04-af0a-ddb34fc205be-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:02.999940 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:02.999908    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"]
Nov 01 06:52:03.004379 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:03.004350    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2260d-689984948c-kzcq4"]
Nov 01 06:52:03.112656 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:03.112577    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"]
Nov 01 06:52:03.118328 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:03.118300    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2260d-56f8767997-h98t8"]
Nov 01 06:52:03.234571 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:03.234539    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6738a9a2-4627-4e04-af0a-ddb34fc205be" path="/var/lib/kubelet/pods/6738a9a2-4627-4e04-af0a-ddb34fc205be/volumes"
Nov 01 06:52:03.234933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:03.234918    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f179c728-3846-4be0-b477-6dc3d9b7ec46" path="/var/lib/kubelet/pods/f179c728-3846-4be0-b477-6dc3d9b7ec46/volumes"
Nov 01 06:52:16.401550 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:16.401470    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:52:17.545739 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:52:17.545701    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:52:27.137790 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:27.137759    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"]
Nov 01 06:52:27.138338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:27.138038    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy" containerID="cri-o://0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8" gracePeriod=30
Nov 01 06:52:32.395431 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.395404    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:52:32.538027 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.537996    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-data\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538048    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-credential-socket\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538103    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c10027c-e632-466c-b086-6fef6fed4292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c10027c-e632-466c-b086-6fef6fed4292-istio-podinfo\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538128    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-envoy\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538145    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-certs\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538213 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538184    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-istio-token\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-istio-token\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538233    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c10027c-e632-466c-b086-6fef6fed4292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c10027c-e632-466c-b086-6fef6fed4292-istiod-ca-cert\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538274    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-data" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:52:32.538506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538325    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-credential-socket" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:52:32.538506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538356    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-socket\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538405    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-kube-api-access-c8nlm\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-kube-api-access-c8nlm\") pod \"0c10027c-e632-466c-b086-6fef6fed4292\" (UID: \"0c10027c-e632-466c-b086-6fef6fed4292\") "
Nov 01 06:52:32.538506 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538481    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-certs" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:52:32.538795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538606    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-socket" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:52:32.538795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538677    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c10027c-e632-466c-b086-6fef6fed4292-istiod-ca-cert" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:32.539004 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538809    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.539004 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538821    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c10027c-e632-466c-b086-6fef6fed4292-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.539004 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538857    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.539004 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538865    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.539004 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.538874    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.540686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.540659    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-istio-token" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:32.540686 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.540678    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-kube-api-access-c8nlm" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "kube-api-access-c8nlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:32.541069 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.541048    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-envoy" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:52:32.541145 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.541079    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c10027c-e632-466c-b086-6fef6fed4292-istio-podinfo" pod "0c10027c-e632-466c-b086-6fef6fed4292" (UID: "0c10027c-e632-466c-b086-6fef6fed4292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:52:32.639689 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.639654    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8nlm\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-kube-api-access-c8nlm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.639689 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.639688    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c10027c-e632-466c-b086-6fef6fed4292-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.639907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.639705    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c10027c-e632-466c-b086-6fef6fed4292-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.639907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.639719    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c10027c-e632-466c-b086-6fef6fed4292-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:32.937685 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.937601    5510 generic.go:356] "Generic (PLEG): container finished" podID="0c10027c-e632-466c-b086-6fef6fed4292" containerID="0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8" exitCode=0
Nov 01 06:52:32.937685 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.937673    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"
Nov 01 06:52:32.937909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.937684    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" event={"ID":"0c10027c-e632-466c-b086-6fef6fed4292","Type":"ContainerDied","Data":"0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8"}
Nov 01 06:52:32.937909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.937766    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf" event={"ID":"0c10027c-e632-466c-b086-6fef6fed4292","Type":"ContainerDied","Data":"1c4ea7bed51cc9658b50a14e6588ba63f28d93aaceab4dbe5989f989c63b5f12"}
Nov 01 06:52:32.937909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.937791    5510 scope.go:122] "RemoveContainer" containerID="0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8"
Nov 01 06:52:32.947985 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.947969    5510 scope.go:122] "RemoveContainer" containerID="a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532"
Nov 01 06:52:32.956394 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.956379    5510 scope.go:122] "RemoveContainer" containerID="0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8"
Nov 01 06:52:32.956635 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:52:32.956618    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8\": container with ID starting with 0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8 not found: ID does not exist" containerID="0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8"
Nov 01 06:52:32.956679 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.956642    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8"} err="failed to get container status \"0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8\": rpc error: code = NotFound desc = could not find container \"0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8\": container with ID starting with 0667c98564d2d8f63bf3076e888ed7f1101a7d4487f4ebe55a93d489a64621c8 not found: ID does not exist"
Nov 01 06:52:32.956679 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.956659    5510 scope.go:122] "RemoveContainer" containerID="a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532"
Nov 01 06:52:32.956885 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:52:32.956864    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532\": container with ID starting with a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532 not found: ID does not exist" containerID="a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532"
Nov 01 06:52:32.956946 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:32.956894    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532"} err="failed to get container status \"a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532\": rpc error: code = NotFound desc = could not find container \"a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532\": container with ID starting with a4f45fa20de1626f78e847c9fb40a434149da73f593f9ced3a78fb00f92ff532 not found: ID does not exist"
Nov 01 06:52:33.101261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:33.101225    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"]
Nov 01 06:52:33.103528 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:33.103504    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2260d-openshift-default-7d8b8c7bfc-bb6rf"]
Nov 01 06:52:33.236536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:33.236506    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c10027c-e632-466c-b086-6fef6fed4292" path="/var/lib/kubelet/pods/0c10027c-e632-466c-b086-6fef6fed4292/volumes"
Nov 01 06:52:35.911639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:35.911610    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"]
Nov 01 06:52:35.912120 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:35.911809    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk" podUID="5364423f-7955-4f92-afb8-f4a54ccd9c0d" containerName="maas-api" containerID="cri-o://6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927" gracePeriod=30
Nov 01 06:52:36.199233 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.199209    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:52:36.280493 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.280467    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5364423f-7955-4f92-afb8-f4a54ccd9c0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5364423f-7955-4f92-afb8-f4a54ccd9c0d-openshift-service-ca\") pod \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") "
Nov 01 06:52:36.280666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.280587    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls\") pod \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") "
Nov 01 06:52:36.280666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.280631    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5364423f-7955-4f92-afb8-f4a54ccd9c0d-kube-api-access-wsc7m\" (UniqueName: \"kubernetes.io/projected/5364423f-7955-4f92-afb8-f4a54ccd9c0d-kube-api-access-wsc7m\") pod \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") "
Nov 01 06:52:36.280666 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.280652    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-metrics-certs\") pod \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\" (UID: \"5364423f-7955-4f92-afb8-f4a54ccd9c0d\") "
Nov 01 06:52:36.280922 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.280900    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5364423f-7955-4f92-afb8-f4a54ccd9c0d-openshift-service-ca" pod "5364423f-7955-4f92-afb8-f4a54ccd9c0d" (UID: "5364423f-7955-4f92-afb8-f4a54ccd9c0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:36.282910 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.282883    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-metrics-certs" pod "5364423f-7955-4f92-afb8-f4a54ccd9c0d" (UID: "5364423f-7955-4f92-afb8-f4a54ccd9c0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:52:36.282910 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.282896    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls" pod "5364423f-7955-4f92-afb8-f4a54ccd9c0d" (UID: "5364423f-7955-4f92-afb8-f4a54ccd9c0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:52:36.283083 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.282942    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5364423f-7955-4f92-afb8-f4a54ccd9c0d-kube-api-access-wsc7m" pod "5364423f-7955-4f92-afb8-f4a54ccd9c0d" (UID: "5364423f-7955-4f92-afb8-f4a54ccd9c0d"). InnerVolumeSpecName "kube-api-access-wsc7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:36.381941 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.381905    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:36.381941 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.381934    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsc7m\" (UniqueName: \"kubernetes.io/projected/5364423f-7955-4f92-afb8-f4a54ccd9c0d-kube-api-access-wsc7m\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:36.381941 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.381945    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5364423f-7955-4f92-afb8-f4a54ccd9c0d-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:36.382174 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.381953    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5364423f-7955-4f92-afb8-f4a54ccd9c0d-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:36.964006 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.963967    5510 generic.go:356] "Generic (PLEG): container finished" podID="5364423f-7955-4f92-afb8-f4a54ccd9c0d" containerID="6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927" exitCode=0
Nov 01 06:52:36.964448 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.964017    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"
Nov 01 06:52:36.964448 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.964047    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk" event={"ID":"5364423f-7955-4f92-afb8-f4a54ccd9c0d","Type":"ContainerDied","Data":"6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927"}
Nov 01 06:52:36.964448 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.964096    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk" event={"ID":"5364423f-7955-4f92-afb8-f4a54ccd9c0d","Type":"ContainerDied","Data":"a88181b6f1c10aac78bbb4f1dfd0f0ac2cd9a003d166c855e86a35996ddcb11b"}
Nov 01 06:52:36.964448 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.964117    5510 scope.go:122] "RemoveContainer" containerID="6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927"
Nov 01 06:52:36.974469 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.974447    5510 scope.go:122] "RemoveContainer" containerID="6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927"
Nov 01 06:52:36.974748 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:52:36.974730    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927\": container with ID starting with 6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927 not found: ID does not exist" containerID="6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927"
Nov 01 06:52:36.974800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:36.974756    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927"} err="failed to get container status \"6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927\": rpc error: code = NotFound desc = could not find container \"6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927\": container with ID starting with 6073f498c59a2eda4961c19d6a841b1a14aa4a27fbaf8bba5cbff33c554c6927 not found: ID does not exist"
Nov 01 06:52:37.126260 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:37.126225    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"]
Nov 01 06:52:37.129944 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:37.129923    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-747b7b-7884dd54bc-k46jk"]
Nov 01 06:52:37.235497 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:37.235418    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5364423f-7955-4f92-afb8-f4a54ccd9c0d" path="/var/lib/kubelet/pods/5364423f-7955-4f92-afb8-f4a54ccd9c0d/volumes"
Nov 01 06:52:44.313048 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:44.313014    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:52:47.318631 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:47.318594    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"]
Nov 01 06:52:47.423637 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:47.423605    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"]
Nov 01 06:52:47.546763 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:52:47.546719    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:52:52.319874 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:52.319809    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs" podUID="9f0555b2-a605-4691-80cd-cdf658ec4f25" containerName="payload-processing" containerID="cri-o://32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33" gracePeriod=25
Nov 01 06:52:52.424019 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:52.423960    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc" podUID="7113c3d8-006a-4f56-8e09-7fc9a16ede18" containerName="payload-pre-processing" containerID="cri-o://2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821" gracePeriod=25
Nov 01 06:52:56.584544 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.584512    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"]
Nov 01 06:52:56.584957 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.584718    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs" podUID="b837176a-c74b-4203-85a9-acb1120c5f97" containerName="maas-api" containerID="cri-o://01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d" gracePeriod=30
Nov 01 06:52:56.850067 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.850006    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:52:56.993374 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.993333    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-metrics-certs\") pod \"b837176a-c74b-4203-85a9-acb1120c5f97\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") "
Nov 01 06:52:56.993563 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.993513    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b837176a-c74b-4203-85a9-acb1120c5f97-kube-api-access-m88jh\" (UniqueName: \"kubernetes.io/projected/b837176a-c74b-4203-85a9-acb1120c5f97-kube-api-access-m88jh\") pod \"b837176a-c74b-4203-85a9-acb1120c5f97\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") "
Nov 01 06:52:56.993650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.993590    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls\") pod \"b837176a-c74b-4203-85a9-acb1120c5f97\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") "
Nov 01 06:52:56.993650 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.993637    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b837176a-c74b-4203-85a9-acb1120c5f97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b837176a-c74b-4203-85a9-acb1120c5f97-openshift-service-ca\") pod \"b837176a-c74b-4203-85a9-acb1120c5f97\" (UID: \"b837176a-c74b-4203-85a9-acb1120c5f97\") "
Nov 01 06:52:56.994257 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.994099    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b837176a-c74b-4203-85a9-acb1120c5f97-openshift-service-ca" pod "b837176a-c74b-4203-85a9-acb1120c5f97" (UID: "b837176a-c74b-4203-85a9-acb1120c5f97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:56.995678 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.995648    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-metrics-certs" pod "b837176a-c74b-4203-85a9-acb1120c5f97" (UID: "b837176a-c74b-4203-85a9-acb1120c5f97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:52:56.995856 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.995812    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls" pod "b837176a-c74b-4203-85a9-acb1120c5f97" (UID: "b837176a-c74b-4203-85a9-acb1120c5f97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:52:56.996057 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:56.996040    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b837176a-c74b-4203-85a9-acb1120c5f97-kube-api-access-m88jh" pod "b837176a-c74b-4203-85a9-acb1120c5f97" (UID: "b837176a-c74b-4203-85a9-acb1120c5f97"). InnerVolumeSpecName "kube-api-access-m88jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:57.071013 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.070984    5510 generic.go:356] "Generic (PLEG): container finished" podID="b837176a-c74b-4203-85a9-acb1120c5f97" containerID="01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d" exitCode=0
Nov 01 06:52:57.071139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.071033    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs" event={"ID":"b837176a-c74b-4203-85a9-acb1120c5f97","Type":"ContainerDied","Data":"01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d"}
Nov 01 06:52:57.071139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.071037    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"
Nov 01 06:52:57.071139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.071074    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs" event={"ID":"b837176a-c74b-4203-85a9-acb1120c5f97","Type":"ContainerDied","Data":"8cec0b8f3eea8884d304a5dc3d5345dde9e7876bbd4a1fb3cc6c28b5b4cc34b2"}
Nov 01 06:52:57.071139 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.071096    5510 scope.go:122] "RemoveContainer" containerID="01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d"
Nov 01 06:52:57.091641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.091603    5510 scope.go:122] "RemoveContainer" containerID="01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d"
Nov 01 06:52:57.092090 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:52:57.092036    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d\": container with ID starting with 01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d not found: ID does not exist" containerID="01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d"
Nov 01 06:52:57.092090 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.092068    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d"} err="failed to get container status \"01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d\": rpc error: code = NotFound desc = could not find container \"01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d\": container with ID starting with 01c2f0baf5860651a5532190150a0e4158cc08a34e64cf81307c964102242e4d not found: ID does not exist"
Nov 01 06:52:57.094584 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.094561    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:57.094705 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.094588    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b837176a-c74b-4203-85a9-acb1120c5f97-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:57.094705 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.094603    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b837176a-c74b-4203-85a9-acb1120c5f97-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:57.094705 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.094615    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m88jh\" (UniqueName: \"kubernetes.io/projected/b837176a-c74b-4203-85a9-acb1120c5f97-kube-api-access-m88jh\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:57.236951 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.236922    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"]
Nov 01 06:52:57.237254 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:57.237233    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71cea4-7ffc899585-l6wbs"]
Nov 01 06:52:59.234784 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:59.234749    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b837176a-c74b-4203-85a9-acb1120c5f97" path="/var/lib/kubelet/pods/b837176a-c74b-4203-85a9-acb1120c5f97/volumes"
Nov 01 06:52:59.714626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:59.714596    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc" podUID="7113c3d8-006a-4f56-8e09-7fc9a16ede18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:59.714801 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:52:59.714598    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs" podUID="9f0555b2-a605-4691-80cd-cdf658ec4f25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:01.163120 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:01.163087    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:53:07.457308 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.457269    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"]
Nov 01 06:53:07.457812 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.457532    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27" podUID="20298942-580f-4a18-8b45-2ed53d03b46d" containerName="maas-api" containerID="cri-o://caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682" gracePeriod=30
Nov 01 06:53:07.714985 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.714928    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:53:07.806268 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.806241    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-metrics-certs\") pod \"20298942-580f-4a18-8b45-2ed53d03b46d\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") "
Nov 01 06:53:07.806439 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.806281    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls\") pod \"20298942-580f-4a18-8b45-2ed53d03b46d\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") "
Nov 01 06:53:07.806439 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.806339    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20298942-580f-4a18-8b45-2ed53d03b46d-kube-api-access-8lfzd\" (UniqueName: \"kubernetes.io/projected/20298942-580f-4a18-8b45-2ed53d03b46d-kube-api-access-8lfzd\") pod \"20298942-580f-4a18-8b45-2ed53d03b46d\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") "
Nov 01 06:53:07.806439 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.806372    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20298942-580f-4a18-8b45-2ed53d03b46d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20298942-580f-4a18-8b45-2ed53d03b46d-openshift-service-ca\") pod \"20298942-580f-4a18-8b45-2ed53d03b46d\" (UID: \"20298942-580f-4a18-8b45-2ed53d03b46d\") "
Nov 01 06:53:07.806781 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.806741    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20298942-580f-4a18-8b45-2ed53d03b46d-openshift-service-ca" pod "20298942-580f-4a18-8b45-2ed53d03b46d" (UID: "20298942-580f-4a18-8b45-2ed53d03b46d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:07.808676 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.808647    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls" pod "20298942-580f-4a18-8b45-2ed53d03b46d" (UID: "20298942-580f-4a18-8b45-2ed53d03b46d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:07.808759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.808670    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20298942-580f-4a18-8b45-2ed53d03b46d-kube-api-access-8lfzd" pod "20298942-580f-4a18-8b45-2ed53d03b46d" (UID: "20298942-580f-4a18-8b45-2ed53d03b46d"). InnerVolumeSpecName "kube-api-access-8lfzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:07.808759 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.808704    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-metrics-certs" pod "20298942-580f-4a18-8b45-2ed53d03b46d" (UID: "20298942-580f-4a18-8b45-2ed53d03b46d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:07.907715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.907680    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:07.907715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.907709    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20298942-580f-4a18-8b45-2ed53d03b46d-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:07.907715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.907719    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lfzd\" (UniqueName: \"kubernetes.io/projected/20298942-580f-4a18-8b45-2ed53d03b46d-kube-api-access-8lfzd\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:07.907988 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:07.907728    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20298942-580f-4a18-8b45-2ed53d03b46d-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:08.130536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.130448    5510 generic.go:356] "Generic (PLEG): container finished" podID="20298942-580f-4a18-8b45-2ed53d03b46d" containerID="caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682" exitCode=0
Nov 01 06:53:08.130536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.130494    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"
Nov 01 06:53:08.130742 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.130530    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27" event={"ID":"20298942-580f-4a18-8b45-2ed53d03b46d","Type":"ContainerDied","Data":"caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682"}
Nov 01 06:53:08.130742 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.130583    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27" event={"ID":"20298942-580f-4a18-8b45-2ed53d03b46d","Type":"ContainerDied","Data":"4a2cbc3e5b331e185a24cfdb2b971af3873f4a84c06383b9e2c19eb985193b5c"}
Nov 01 06:53:08.130742 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.130606    5510 scope.go:122] "RemoveContainer" containerID="caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682"
Nov 01 06:53:08.141289 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.141272    5510 scope.go:122] "RemoveContainer" containerID="caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682"
Nov 01 06:53:08.141542 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:53:08.141523    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682\": container with ID starting with caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682 not found: ID does not exist" containerID="caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682"
Nov 01 06:53:08.141599 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.141551    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682"} err="failed to get container status \"caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682\": rpc error: code = NotFound desc = could not find container \"caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682\": container with ID starting with caa4ac4521f81409f3c48d8e1cda5d7a138f7e2ba3257a0dbc815db8b47c4682 not found: ID does not exist"
Nov 01 06:53:08.293836 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.293806    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"]
Nov 01 06:53:08.297709 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:08.297682    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cfe368-86d97f6d5f-5wt27"]
Nov 01 06:53:09.235294 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:09.235255    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20298942-580f-4a18-8b45-2ed53d03b46d" path="/var/lib/kubelet/pods/20298942-580f-4a18-8b45-2ed53d03b46d/volumes"
Nov 01 06:53:09.714578 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:09.714544    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc" podUID="7113c3d8-006a-4f56-8e09-7fc9a16ede18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:09.714743 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:09.714549    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs" podUID="9f0555b2-a605-4691-80cd-cdf658ec4f25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:12.570964 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.570936    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:53:12.656443 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.656413    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0555b2-a605-4691-80cd-cdf658ec4f25-kube-api-access-67vp4\" (UniqueName: \"kubernetes.io/projected/9f0555b2-a605-4691-80cd-cdf658ec4f25-kube-api-access-67vp4\") pod \"9f0555b2-a605-4691-80cd-cdf658ec4f25\" (UID: \"9f0555b2-a605-4691-80cd-cdf658ec4f25\") "
Nov 01 06:53:12.656640 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.656525    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0555b2-a605-4691-80cd-cdf658ec4f25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0555b2-a605-4691-80cd-cdf658ec4f25-plugins-config-volume\") pod \"9f0555b2-a605-4691-80cd-cdf658ec4f25\" (UID: \"9f0555b2-a605-4691-80cd-cdf658ec4f25\") "
Nov 01 06:53:12.656944 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.656924    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0555b2-a605-4691-80cd-cdf658ec4f25-plugins-config-volume" pod "9f0555b2-a605-4691-80cd-cdf658ec4f25" (UID: "9f0555b2-a605-4691-80cd-cdf658ec4f25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:12.658715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.658692    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0555b2-a605-4691-80cd-cdf658ec4f25-kube-api-access-67vp4" pod "9f0555b2-a605-4691-80cd-cdf658ec4f25" (UID: "9f0555b2-a605-4691-80cd-cdf658ec4f25"). InnerVolumeSpecName "kube-api-access-67vp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:12.682651 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.682627    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:53:12.757449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.757412    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7113c3d8-006a-4f56-8e09-7fc9a16ede18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7113c3d8-006a-4f56-8e09-7fc9a16ede18-plugins-config-volume\") pod \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\" (UID: \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\") "
Nov 01 06:53:12.757449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.757466    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7113c3d8-006a-4f56-8e09-7fc9a16ede18-kube-api-access-mtrss\" (UniqueName: \"kubernetes.io/projected/7113c3d8-006a-4f56-8e09-7fc9a16ede18-kube-api-access-mtrss\") pod \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\" (UID: \"7113c3d8-006a-4f56-8e09-7fc9a16ede18\") "
Nov 01 06:53:12.757668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.757630    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67vp4\" (UniqueName: \"kubernetes.io/projected/9f0555b2-a605-4691-80cd-cdf658ec4f25-kube-api-access-67vp4\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:12.757668 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.757641    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0555b2-a605-4691-80cd-cdf658ec4f25-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:12.757887 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.757863    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7113c3d8-006a-4f56-8e09-7fc9a16ede18-plugins-config-volume" pod "7113c3d8-006a-4f56-8e09-7fc9a16ede18" (UID: "7113c3d8-006a-4f56-8e09-7fc9a16ede18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:12.759702 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.759686    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7113c3d8-006a-4f56-8e09-7fc9a16ede18-kube-api-access-mtrss" pod "7113c3d8-006a-4f56-8e09-7fc9a16ede18" (UID: "7113c3d8-006a-4f56-8e09-7fc9a16ede18"). InnerVolumeSpecName "kube-api-access-mtrss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:12.859282 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.859193    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7113c3d8-006a-4f56-8e09-7fc9a16ede18-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:12.859282 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:12.859233    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtrss\" (UniqueName: \"kubernetes.io/projected/7113c3d8-006a-4f56-8e09-7fc9a16ede18-kube-api-access-mtrss\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:13.085217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.085178    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:53:13.158866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.158755    5510 generic.go:356] "Generic (PLEG): container finished" podID="7113c3d8-006a-4f56-8e09-7fc9a16ede18" containerID="2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821" exitCode=0
Nov 01 06:53:13.158866 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.158810    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"
Nov 01 06:53:13.159093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.158864    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc" event={"ID":"7113c3d8-006a-4f56-8e09-7fc9a16ede18","Type":"ContainerDied","Data":"2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821"}
Nov 01 06:53:13.159093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.158924    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc" event={"ID":"7113c3d8-006a-4f56-8e09-7fc9a16ede18","Type":"ContainerDied","Data":"04c5f192f33ce232b35b9c05b6def23f4db465473073473f73c46dabb4027a2f"}
Nov 01 06:53:13.159093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.158957    5510 scope.go:122] "RemoveContainer" containerID="2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821"
Nov 01 06:53:13.160313 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.160287    5510 generic.go:356] "Generic (PLEG): container finished" podID="9f0555b2-a605-4691-80cd-cdf658ec4f25" containerID="32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33" exitCode=0
Nov 01 06:53:13.160449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.160323    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs" event={"ID":"9f0555b2-a605-4691-80cd-cdf658ec4f25","Type":"ContainerDied","Data":"32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33"}
Nov 01 06:53:13.160449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.160331    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"
Nov 01 06:53:13.160449 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.160361    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs" event={"ID":"9f0555b2-a605-4691-80cd-cdf658ec4f25","Type":"ContainerDied","Data":"b53069dde74a94bc6b88c36b1d4b79150f3d6ab6bcf91c937efe4c707e7977d9"}
Nov 01 06:53:13.169207 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.169189    5510 scope.go:122] "RemoveContainer" containerID="2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821"
Nov 01 06:53:13.169483 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:53:13.169465    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821\": container with ID starting with 2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821 not found: ID does not exist" containerID="2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821"
Nov 01 06:53:13.169568 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.169494    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821"} err="failed to get container status \"2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821\": rpc error: code = NotFound desc = could not find container \"2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821\": container with ID starting with 2df579975557376489f7ac1c8b7d264c3c48e4394a1ae4cc35b570c8bc982821 not found: ID does not exist"
Nov 01 06:53:13.169568 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.169516    5510 scope.go:122] "RemoveContainer" containerID="32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33"
Nov 01 06:53:13.177549 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.177524    5510 scope.go:122] "RemoveContainer" containerID="32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33"
Nov 01 06:53:13.177848 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:53:13.177816    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33\": container with ID starting with 32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33 not found: ID does not exist" containerID="32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33"
Nov 01 06:53:13.177912 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.177856    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33"} err="failed to get container status \"32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33\": rpc error: code = NotFound desc = could not find container \"32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33\": container with ID starting with 32c66315f95ad7475ae95fbc46cb784a1dde2abc6acdb9ee4f5fb55dff8ced33 not found: ID does not exist"
Nov 01 06:53:13.330157 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.330114    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"]
Nov 01 06:53:13.330157 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.330165    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cfe368-65c658b5cf-cr7vc"]
Nov 01 06:53:13.436905 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.436796    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"]
Nov 01 06:53:13.442081 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:13.442054    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cfe368-7547d47ddf-vddxs"]
Nov 01 06:53:15.234682 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:15.234647    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7113c3d8-006a-4f56-8e09-7fc9a16ede18" path="/var/lib/kubelet/pods/7113c3d8-006a-4f56-8e09-7fc9a16ede18/volumes"
Nov 01 06:53:15.235160 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:15.235144    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0555b2-a605-4691-80cd-cdf658ec4f25" path="/var/lib/kubelet/pods/9f0555b2-a605-4691-80cd-cdf658ec4f25/volumes"
Nov 01 06:53:17.544175 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:53:17.544127    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:53:28.909003 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:28.908971    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:53:36.567912 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:36.567877    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"]
Nov 01 06:53:36.670768 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:36.670735    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"]
Nov 01 06:53:37.100378 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100341    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"]
Nov 01 06:53:37.100763 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100748    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy"
Nov 01 06:53:37.100819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100763    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3b79fb-4201-4024-b2d4-fdeb139fef14" containerName="istio-proxy"
Nov 01 06:53:37.100819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100777    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cfce2fb8-cb98-490e-8b04-6e2a45e2910e" containerName="payload-processing"
Nov 01 06:53:37.100819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100782    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfce2fb8-cb98-490e-8b04-6e2a45e2910e" containerName="payload-processing"
Nov 01 06:53:37.100819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100791    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy"
Nov 01 06:53:37.100819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100796    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy"
Nov 01 06:53:37.100819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100807    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy"
Nov 01 06:53:37.100819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100812    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy"
Nov 01 06:53:37.100819 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100821    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f179c728-3846-4be0-b477-6dc3d9b7ec46" containerName="payload-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100841    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f179c728-3846-4be0-b477-6dc3d9b7ec46" containerName="payload-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100859    5510 cpu_manager.go:414] "containerMap: removing container" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100864    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100870    5510 cpu_manager.go:414] "containerMap: removing container" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100874    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100883    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6738a9a2-4627-4e04-af0a-ddb34fc205be" containerName="payload-pre-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100889    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6738a9a2-4627-4e04-af0a-ddb34fc205be" containerName="payload-pre-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100899    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a82ed2f5-f019-4457-a3f9-b99da85c104f" containerName="payload-pre-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100904    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82ed2f5-f019-4457-a3f9-b99da85c104f" containerName="payload-pre-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100915    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b837176a-c74b-4203-85a9-acb1120c5f97" containerName="maas-api"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100919    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b837176a-c74b-4203-85a9-acb1120c5f97" containerName="maas-api"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100925    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9f0555b2-a605-4691-80cd-cdf658ec4f25" containerName="payload-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100930    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0555b2-a605-4691-80cd-cdf658ec4f25" containerName="payload-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100937    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5364423f-7955-4f92-afb8-f4a54ccd9c0d" containerName="maas-api"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100942    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5364423f-7955-4f92-afb8-f4a54ccd9c0d" containerName="maas-api"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100950    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e7176c24-60a4-4250-822f-84fd0b79b881" containerName="payload-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100955    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7176c24-60a4-4250-822f-84fd0b79b881" containerName="payload-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100961    5510 cpu_manager.go:414] "containerMap: removing container" podUID="20298942-580f-4a18-8b45-2ed53d03b46d" containerName="maas-api"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.100965    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20298942-580f-4a18-8b45-2ed53d03b46d" containerName="maas-api"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101005    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7113c3d8-006a-4f56-8e09-7fc9a16ede18" containerName="payload-pre-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101010    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7113c3d8-006a-4f56-8e09-7fc9a16ede18" containerName="payload-pre-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101115    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7113c3d8-006a-4f56-8e09-7fc9a16ede18" containerName="payload-pre-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101132    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfce2fb8-cb98-490e-8b04-6e2a45e2910e" containerName="payload-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101142    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101152    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b571d511-15fb-4937-bc74-2c808ab8fec3" containerName="istio-proxy"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101163    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f179c728-3846-4be0-b477-6dc3d9b7ec46" containerName="payload-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101176    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101186    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="75de03a3-8add-411c-a293-9487c9b218ad" containerName="istio-proxy"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101196    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6738a9a2-4627-4e04-af0a-ddb34fc205be" containerName="payload-pre-processing"
Nov 01 06:53:37.101193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101207    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82ed2f5-f019-4457-a3f9-b99da85c104f" containerName="payload-pre-processing"
Nov 01 06:53:37.102137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101217    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b837176a-c74b-4203-85a9-acb1120c5f97" containerName="maas-api"
Nov 01 06:53:37.102137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101226    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy"
Nov 01 06:53:37.102137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101234    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0555b2-a605-4691-80cd-cdf658ec4f25" containerName="payload-processing"
Nov 01 06:53:37.102137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101241    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5364423f-7955-4f92-afb8-f4a54ccd9c0d" containerName="maas-api"
Nov 01 06:53:37.102137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101250    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7176c24-60a4-4250-822f-84fd0b79b881" containerName="payload-processing"
Nov 01 06:53:37.102137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.101259    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="20298942-580f-4a18-8b45-2ed53d03b46d" containerName="maas-api"
Nov 01 06:53:37.105687 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.105671    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.160691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.160664    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:53:37.160855 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.160788    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:53:37.160959 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.160868    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:53:37.160959 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.160880    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:53:37.163113 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.163093    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"]
Nov 01 06:53:37.202441 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.202414    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.202611 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.202488    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.202611 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.202543    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhg4\" (UniqueName: \"kubernetes.io/projected/78bb2b36-be1a-4503-9a20-af256f5c6dd9-kube-api-access-pzhg4\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.303565 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.303537    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.303766 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.303623    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.303766 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.303640    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhg4\" (UniqueName: \"kubernetes.io/projected/78bb2b36-be1a-4503-9a20-af256f5c6dd9-kube-api-access-pzhg4\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.304358 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.304332    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.306176 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.306156    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.364870 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.364776    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhg4\" (UniqueName: \"kubernetes.io/projected/78bb2b36-be1a-4503-9a20-af256f5c6dd9-kube-api-access-pzhg4\") pod \"maas-api-revoke-sub-8f54b64a-v56kg\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.418799 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.418770    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:37.788795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:37.788764    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"]
Nov 01 06:53:37.792040 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:53:37.792015    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bb2b36_be1a_4503_9a20_af256f5c6dd9.slice/crio-fd01883fe4bc820713d8e8ab33e7df1da3f29d1f5219cecf0b54e2792fa30dcb WatchSource:0}: Error finding container fd01883fe4bc820713d8e8ab33e7df1da3f29d1f5219cecf0b54e2792fa30dcb: Status 404 returned error can't find the container with id fd01883fe4bc820713d8e8ab33e7df1da3f29d1f5219cecf0b54e2792fa30dcb
Nov 01 06:53:38.284436 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:38.284395    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg" event={"ID":"78bb2b36-be1a-4503-9a20-af256f5c6dd9","Type":"ContainerStarted","Data":"a8511a15a3bdf8f6a5b4ad55de7737ffa7cf2aac47414c5b5460f4603c3a4aa9"}
Nov 01 06:53:38.284600 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:38.284452    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg" event={"ID":"78bb2b36-be1a-4503-9a20-af256f5c6dd9","Type":"ContainerStarted","Data":"fd01883fe4bc820713d8e8ab33e7df1da3f29d1f5219cecf0b54e2792fa30dcb"}
Nov 01 06:53:38.399734 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:38.399624    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg" podStartSLOduration=1.399611872 podStartE2EDuration="1.399611872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:53:38.399060421 +0000 UTC m=+4055.277033831" watchObservedRunningTime="2026-11-01 06:53:38.399611872 +0000 UTC m=+4055.277585338"
Nov 01 06:53:39.290875 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:39.290822    5510 generic.go:356] "Generic (PLEG): container finished" podID="78bb2b36-be1a-4503-9a20-af256f5c6dd9" containerID="a8511a15a3bdf8f6a5b4ad55de7737ffa7cf2aac47414c5b5460f4603c3a4aa9" exitCode=0
Nov 01 06:53:39.291295 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:39.290920    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg" event={"ID":"78bb2b36-be1a-4503-9a20-af256f5c6dd9","Type":"ContainerDied","Data":"a8511a15a3bdf8f6a5b4ad55de7737ffa7cf2aac47414c5b5460f4603c3a4aa9"}
Nov 01 06:53:40.446152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.446130    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:40.533278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.533244    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78bb2b36-be1a-4503-9a20-af256f5c6dd9-kube-api-access-pzhg4\" (UniqueName: \"kubernetes.io/projected/78bb2b36-be1a-4503-9a20-af256f5c6dd9-kube-api-access-pzhg4\") pod \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") "
Nov 01 06:53:40.533431 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.533345    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-xks-ca\") pod \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") "
Nov 01 06:53:40.533431 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.533397    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-openshift-ca\") pod \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\" (UID: \"78bb2b36-be1a-4503-9a20-af256f5c6dd9\") "
Nov 01 06:53:40.533716 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.533692    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-openshift-ca" pod "78bb2b36-be1a-4503-9a20-af256f5c6dd9" (UID: "78bb2b36-be1a-4503-9a20-af256f5c6dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:40.535489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.535470    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-xks-ca" pod "78bb2b36-be1a-4503-9a20-af256f5c6dd9" (UID: "78bb2b36-be1a-4503-9a20-af256f5c6dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:40.535609 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.535591    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bb2b36-be1a-4503-9a20-af256f5c6dd9-kube-api-access-pzhg4" pod "78bb2b36-be1a-4503-9a20-af256f5c6dd9" (UID: "78bb2b36-be1a-4503-9a20-af256f5c6dd9"). InnerVolumeSpecName "kube-api-access-pzhg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:40.634374 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.634291    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhg4\" (UniqueName: \"kubernetes.io/projected/78bb2b36-be1a-4503-9a20-af256f5c6dd9-kube-api-access-pzhg4\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:40.634374 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.634319    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:40.634374 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:40.634330    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bb2b36-be1a-4503-9a20-af256f5c6dd9-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.300928 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:41.300896    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"
Nov 01 06:53:41.301097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:41.300899    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg" event={"ID":"78bb2b36-be1a-4503-9a20-af256f5c6dd9","Type":"ContainerDied","Data":"fd01883fe4bc820713d8e8ab33e7df1da3f29d1f5219cecf0b54e2792fa30dcb"}
Nov 01 06:53:41.301097 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:41.301024    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd01883fe4bc820713d8e8ab33e7df1da3f29d1f5219cecf0b54e2792fa30dcb"
Nov 01 06:53:41.446111 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:41.446076    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"]
Nov 01 06:53:41.494316 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:41.494241    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"]
Nov 01 06:53:41.568851 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:41.568782    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp" podUID="c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" containerName="payload-processing" containerID="cri-o://221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0" gracePeriod=25
Nov 01 06:53:41.671394 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:41.671349    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv" podUID="f48ed9e9-7396-4f40-8074-01195481a5af" containerName="payload-pre-processing" containerID="cri-o://76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1" gracePeriod=25
Nov 01 06:53:47.291672 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:47.291592    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp" podUID="c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:47.292070 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:47.291593    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv" podUID="f48ed9e9-7396-4f40-8074-01195481a5af" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:47.544264 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:53:47.544170    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:53:48.600045 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.600009    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"]
Nov 01 06:53:48.600450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.600437    5510 cpu_manager.go:414] "containerMap: removing container" podUID="78bb2b36-be1a-4503-9a20-af256f5c6dd9" containerName="revoke-keys"
Nov 01 06:53:48.600494 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.600451    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78bb2b36-be1a-4503-9a20-af256f5c6dd9" containerName="revoke-keys"
Nov 01 06:53:48.600494 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.600482    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy"
Nov 01 06:53:48.600494 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.600487    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c10027c-e632-466c-b086-6fef6fed4292" containerName="istio-proxy"
Nov 01 06:53:48.600595 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.600549    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="78bb2b36-be1a-4503-9a20-af256f5c6dd9" containerName="revoke-keys"
Nov 01 06:53:48.603771 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.603757    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.655691 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.655671    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:53:48.705278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.705252    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:53:48.705438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.705321    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:53:48.705438 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.705376    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:53:48.707135 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.707112    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"]
Nov 01 06:53:48.711800 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.711781    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.711863 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.711812    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6lw\" (UniqueName: \"kubernetes.io/projected/aec10f38-d017-4b23-9974-6a5ee329c0b6-kube-api-access-9f6lw\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.711966 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.711954    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.812996 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.812965    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.813224 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.813083    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.813224 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.813120    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6lw\" (UniqueName: \"kubernetes.io/projected/aec10f38-d017-4b23-9974-6a5ee329c0b6-kube-api-access-9f6lw\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.814021 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.813999    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.816413 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.816386    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.873981 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.873923    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6lw\" (UniqueName: \"kubernetes.io/projected/aec10f38-d017-4b23-9974-6a5ee329c0b6-kube-api-access-9f6lw\") pod \"maas-api-revoke-sub-12e876bf-zxc5c\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:48.916453 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:48.916425    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:49.288679 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:49.288654    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"]
Nov 01 06:53:49.291368 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:53:49.291343    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec10f38_d017_4b23_9974_6a5ee329c0b6.slice/crio-26f12ac3795b5c709bd3b367999ee465b423b5378a10e97d5d66743e9ee012c0 WatchSource:0}: Error finding container 26f12ac3795b5c709bd3b367999ee465b423b5378a10e97d5d66743e9ee012c0: Status 404 returned error can't find the container with id 26f12ac3795b5c709bd3b367999ee465b423b5378a10e97d5d66743e9ee012c0
Nov 01 06:53:49.307179 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:49.307155    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:53:49.340921 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:49.340891    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c" event={"ID":"aec10f38-d017-4b23-9974-6a5ee329c0b6","Type":"ContainerStarted","Data":"26f12ac3795b5c709bd3b367999ee465b423b5378a10e97d5d66743e9ee012c0"}
Nov 01 06:53:50.259348 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:50.259318    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"]
Nov 01 06:53:50.347615 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:50.347586    5510 generic.go:356] "Generic (PLEG): container finished" podID="aec10f38-d017-4b23-9974-6a5ee329c0b6" containerID="5a768ff31b104ba1b98a7a68588f8cceb9bfe5cc5edc7c3dce240f1599484bd0" exitCode=0
Nov 01 06:53:50.347786 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:50.347635    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c" event={"ID":"aec10f38-d017-4b23-9974-6a5ee329c0b6","Type":"ContainerDied","Data":"5a768ff31b104ba1b98a7a68588f8cceb9bfe5cc5edc7c3dce240f1599484bd0"}
Nov 01 06:53:50.369070 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:50.369042    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"]
Nov 01 06:53:51.498437 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.498411    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:51.549037 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.549003    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-xks-ca\") pod \"aec10f38-d017-4b23-9974-6a5ee329c0b6\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") "
Nov 01 06:53:51.549195 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.549122    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-openshift-ca\") pod \"aec10f38-d017-4b23-9974-6a5ee329c0b6\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") "
Nov 01 06:53:51.549263 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.549207    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec10f38-d017-4b23-9974-6a5ee329c0b6-kube-api-access-9f6lw\" (UniqueName: \"kubernetes.io/projected/aec10f38-d017-4b23-9974-6a5ee329c0b6-kube-api-access-9f6lw\") pod \"aec10f38-d017-4b23-9974-6a5ee329c0b6\" (UID: \"aec10f38-d017-4b23-9974-6a5ee329c0b6\") "
Nov 01 06:53:51.549457 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.549434    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-openshift-ca" pod "aec10f38-d017-4b23-9974-6a5ee329c0b6" (UID: "aec10f38-d017-4b23-9974-6a5ee329c0b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:51.549556 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.549542    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:51.551307 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.551286    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec10f38-d017-4b23-9974-6a5ee329c0b6-kube-api-access-9f6lw" pod "aec10f38-d017-4b23-9974-6a5ee329c0b6" (UID: "aec10f38-d017-4b23-9974-6a5ee329c0b6"). InnerVolumeSpecName "kube-api-access-9f6lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:51.551382 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.551363    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-xks-ca" pod "aec10f38-d017-4b23-9974-6a5ee329c0b6" (UID: "aec10f38-d017-4b23-9974-6a5ee329c0b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:51.651956 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.651853    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6lw\" (UniqueName: \"kubernetes.io/projected/aec10f38-d017-4b23-9974-6a5ee329c0b6-kube-api-access-9f6lw\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:51.651956 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:51.651888    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec10f38-d017-4b23-9974-6a5ee329c0b6-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:52.359432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:52.359407    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"
Nov 01 06:53:52.359607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:52.359406    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c" event={"ID":"aec10f38-d017-4b23-9974-6a5ee329c0b6","Type":"ContainerDied","Data":"26f12ac3795b5c709bd3b367999ee465b423b5378a10e97d5d66743e9ee012c0"}
Nov 01 06:53:52.359607 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:52.359526    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f12ac3795b5c709bd3b367999ee465b423b5378a10e97d5d66743e9ee012c0"
Nov 01 06:53:55.260227 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:55.260169    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m" podUID="357950be-1cc3-4216-95a9-376698aebd91" containerName="payload-processing" containerID="cri-o://2e34985c8f92d54b7d23fd784cbb2a588223e01b1001e96b3c41266fadca465a" gracePeriod=25
Nov 01 06:53:55.369947 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:55.369863    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds" podUID="4e157229-ba78-4611-91b8-270c29c0dcdc" containerName="payload-pre-processing" containerID="cri-o://84c5f81542394ffdf83fe73683db492ace80ca8db8cdeab51c16652ac8e332ea" gracePeriod=25
Nov 01 06:53:56.244203 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.244168    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m" podUID="357950be-1cc3-4216-95a9-376698aebd91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:56.467146 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.467114    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7" podUID="ccd4246d-28af-4510-a7cd-d02e63d6fb8b" containerName="main" containerID="cri-o://21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2" gracePeriod=45
Nov 01 06:53:56.513646 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.513585    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6" podUID="5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" containerName="main" containerID="cri-o://58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33" gracePeriod=45
Nov 01 06:53:56.634208 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.634175    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7" podUID="ccd4246d-28af-4510-a7cd-d02e63d6fb8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused"
Nov 01 06:53:56.778187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.778134    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 06:53:56.801923 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.801901    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 06:53:56.899704 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.899667    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-model-cache\") pod \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") "
Nov 01 06:53:56.899865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.899756    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tmp-dir\") pod \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") "
Nov 01 06:53:56.899865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.899791    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tls-certs\") pod \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") "
Nov 01 06:53:56.899865 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.899842    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tls-certs\") pod \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") "
Nov 01 06:53:56.900064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.899889    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-dshm\") pod \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") "
Nov 01 06:53:56.900064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.899954    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-kube-api-access-p9skg\" (UniqueName: \"kubernetes.io/projected/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-kube-api-access-p9skg\") pod \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") "
Nov 01 06:53:56.900064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.899987    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-dshm\") pod \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") "
Nov 01 06:53:56.900064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900001    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tmp-dir" pod "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" (UID: "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:56.900064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900013    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tmp-dir\") pod \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") "
Nov 01 06:53:56.900355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900093    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-model-cache\") pod \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") "
Nov 01 06:53:56.900355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900134    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-home\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-home\") pod \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\" (UID: \"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9\") "
Nov 01 06:53:56.900355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900165    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-home\") pod \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") "
Nov 01 06:53:56.900355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900225    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-kube-api-access-mqmr5\" (UniqueName: \"kubernetes.io/projected/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-kube-api-access-mqmr5\") pod \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\" (UID: \"ccd4246d-28af-4510-a7cd-d02e63d6fb8b\") "
Nov 01 06:53:56.900355 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900321    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tmp-dir" pod "ccd4246d-28af-4510-a7cd-d02e63d6fb8b" (UID: "ccd4246d-28af-4510-a7cd-d02e63d6fb8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:56.900740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900530    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-home" pod "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" (UID: "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:56.900740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900570    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-home" pod "ccd4246d-28af-4510-a7cd-d02e63d6fb8b" (UID: "ccd4246d-28af-4510-a7cd-d02e63d6fb8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:56.900740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900724    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tmp-dir\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:56.900740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900744    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-home\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:56.900740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900758    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-home\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:56.900740 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900772    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tmp-dir\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:56.901172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.900817    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-model-cache" pod "ccd4246d-28af-4510-a7cd-d02e63d6fb8b" (UID: "ccd4246d-28af-4510-a7cd-d02e63d6fb8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:56.901172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.901020    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-model-cache" pod "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" (UID: "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:56.902930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.902890    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tls-certs" pod "ccd4246d-28af-4510-a7cd-d02e63d6fb8b" (UID: "ccd4246d-28af-4510-a7cd-d02e63d6fb8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:56.902930 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.902917    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tls-certs" pod "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" (UID: "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:56.903328 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.903303    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-kube-api-access-mqmr5" pod "ccd4246d-28af-4510-a7cd-d02e63d6fb8b" (UID: "ccd4246d-28af-4510-a7cd-d02e63d6fb8b"). InnerVolumeSpecName "kube-api-access-mqmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:56.903502 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.903340    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-kube-api-access-p9skg" pod "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" (UID: "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9"). InnerVolumeSpecName "kube-api-access-p9skg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:56.903568 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.903514    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-dshm" pod "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" (UID: "5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:56.903665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:56.903647    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-dshm" pod "ccd4246d-28af-4510-a7cd-d02e63d6fb8b" (UID: "ccd4246d-28af-4510-a7cd-d02e63d6fb8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:57.001723 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.001670    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-model-cache\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:57.001723 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.001707    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-tls-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:57.001723 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.001724    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-tls-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:57.002051 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.001739    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-dshm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:57.002051 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.001755    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9skg\" (UniqueName: \"kubernetes.io/projected/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-kube-api-access-p9skg\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:57.002051 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.001769    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9-dshm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:57.002051 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.001786    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-model-cache\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:57.002051 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.001801    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqmr5\" (UniqueName: \"kubernetes.io/projected/ccd4246d-28af-4510-a7cd-d02e63d6fb8b-kube-api-access-mqmr5\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:57.291403 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.291365    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv" podUID="f48ed9e9-7396-4f40-8074-01195481a5af" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:57.291533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.291377    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp" podUID="c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:57.392191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.392152    5510 generic.go:356] "Generic (PLEG): container finished" podID="ccd4246d-28af-4510-a7cd-d02e63d6fb8b" containerID="21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2" exitCode=0
Nov 01 06:53:57.392362 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.392203    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"
Nov 01 06:53:57.392362 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.392242    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7" event={"ID":"ccd4246d-28af-4510-a7cd-d02e63d6fb8b","Type":"ContainerDied","Data":"21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2"}
Nov 01 06:53:57.392362 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.392296    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7" event={"ID":"ccd4246d-28af-4510-a7cd-d02e63d6fb8b","Type":"ContainerDied","Data":"a9ef0f23671a45594732a59be009f1a79b88864f9f01ba6d973e6f7975af6541"}
Nov 01 06:53:57.392362 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.392321    5510 scope.go:122] "RemoveContainer" containerID="21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2"
Nov 01 06:53:57.394569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.393967    5510 generic.go:356] "Generic (PLEG): container finished" podID="5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" containerID="58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33" exitCode=0
Nov 01 06:53:57.394569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.394012    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6" event={"ID":"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9","Type":"ContainerDied","Data":"58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33"}
Nov 01 06:53:57.394569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.394044    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6" event={"ID":"5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9","Type":"ContainerDied","Data":"d7e450006d493fbc73ea751b546b8b0f5e927ed6f99e5e8d1854ed33d6c02dfe"}
Nov 01 06:53:57.394569 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.394020    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"
Nov 01 06:53:57.402372 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.402350    5510 scope.go:122] "RemoveContainer" containerID="21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2"
Nov 01 06:53:57.402656 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:53:57.402630    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2\": container with ID starting with 21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2 not found: ID does not exist" containerID="21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2"
Nov 01 06:53:57.402715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.402664    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2"} err="failed to get container status \"21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2\": rpc error: code = NotFound desc = could not find container \"21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2\": container with ID starting with 21de036a99a0e891fa4f9e56becbe67393ba546f7719ff57884e0221445650a2 not found: ID does not exist"
Nov 01 06:53:57.402715 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.402679    5510 scope.go:122] "RemoveContainer" containerID="58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33"
Nov 01 06:53:57.410774 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.410687    5510 scope.go:122] "RemoveContainer" containerID="58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33"
Nov 01 06:53:57.411055 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:53:57.411030    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33\": container with ID starting with 58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33 not found: ID does not exist" containerID="58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33"
Nov 01 06:53:57.411144 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.411067    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33"} err="failed to get container status \"58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33\": rpc error: code = NotFound desc = could not find container \"58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33\": container with ID starting with 58e7c149e580f6a8f17340beef420c710d157918b48b8148c1d0488ee9d64f33 not found: ID does not exist"
Nov 01 06:53:57.555172 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.555097    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"]
Nov 01 06:53:57.558862 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.558812    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/premium-w0-9c8fc5-kserve-5cdf58dfd8-bkml7"]
Nov 01 06:53:57.666306 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.666278    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"]
Nov 01 06:53:57.673403 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:57.673377    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9c8fc5/simulator-w0-9c8fc5-kserve-86f756d849-brfz6"]
Nov 01 06:53:59.233858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:59.233811    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" path="/var/lib/kubelet/pods/5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9/volumes"
Nov 01 06:53:59.234245 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:59.234216    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd4246d-28af-4510-a7cd-d02e63d6fb8b" path="/var/lib/kubelet/pods/ccd4246d-28af-4510-a7cd-d02e63d6fb8b/volumes"
Nov 01 06:53:59.865498 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:59.865455    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"]
Nov 01 06:53:59.865700 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:53:59.865677    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy" containerID="cri-o://a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1" gracePeriod=30
Nov 01 06:54:01.829119 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:01.829050    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:54:01.916953 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:01.916930    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:54:01.952614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:01.952579    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-plugins-config-volume\") pod \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\" (UID: \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\") "
Nov 01 06:54:01.952795 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:01.952735    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-kube-api-access-xjc55\" (UniqueName: \"kubernetes.io/projected/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-kube-api-access-xjc55\") pod \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\" (UID: \"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8\") "
Nov 01 06:54:01.953002 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:01.952977    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-plugins-config-volume" pod "c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" (UID: "c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:01.954978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:01.954953    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-kube-api-access-xjc55" pod "c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" (UID: "c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8"). InnerVolumeSpecName "kube-api-access-xjc55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:02.053848 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.053792    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48ed9e9-7396-4f40-8074-01195481a5af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48ed9e9-7396-4f40-8074-01195481a5af-plugins-config-volume\") pod \"f48ed9e9-7396-4f40-8074-01195481a5af\" (UID: \"f48ed9e9-7396-4f40-8074-01195481a5af\") "
Nov 01 06:54:02.054015 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.053911    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48ed9e9-7396-4f40-8074-01195481a5af-kube-api-access-l5xj4\" (UniqueName: \"kubernetes.io/projected/f48ed9e9-7396-4f40-8074-01195481a5af-kube-api-access-l5xj4\") pod \"f48ed9e9-7396-4f40-8074-01195481a5af\" (UID: \"f48ed9e9-7396-4f40-8074-01195481a5af\") "
Nov 01 06:54:02.054182 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.054157    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48ed9e9-7396-4f40-8074-01195481a5af-plugins-config-volume" pod "f48ed9e9-7396-4f40-8074-01195481a5af" (UID: "f48ed9e9-7396-4f40-8074-01195481a5af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:02.054261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.054180    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjc55\" (UniqueName: \"kubernetes.io/projected/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-kube-api-access-xjc55\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:02.054261 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.054194    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:02.056137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.056116    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48ed9e9-7396-4f40-8074-01195481a5af-kube-api-access-l5xj4" pod "f48ed9e9-7396-4f40-8074-01195481a5af" (UID: "f48ed9e9-7396-4f40-8074-01195481a5af"). InnerVolumeSpecName "kube-api-access-l5xj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:02.155095 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.155023    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48ed9e9-7396-4f40-8074-01195481a5af-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:02.155095 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.155051    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xj4\" (UniqueName: \"kubernetes.io/projected/f48ed9e9-7396-4f40-8074-01195481a5af-kube-api-access-l5xj4\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:02.421614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.421522    5510 generic.go:356] "Generic (PLEG): container finished" podID="c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" containerID="221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0" exitCode=0
Nov 01 06:54:02.421614 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.421584    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"
Nov 01 06:54:02.421862 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.421615    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp" event={"ID":"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8","Type":"ContainerDied","Data":"221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0"}
Nov 01 06:54:02.421862 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.421668    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp" event={"ID":"c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8","Type":"ContainerDied","Data":"032cc35df92c47046a70aab67c059bdbac87d3b59418c745ced2b22c33fa7b4f"}
Nov 01 06:54:02.421862 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.421690    5510 scope.go:122] "RemoveContainer" containerID="221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0"
Nov 01 06:54:02.423244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.423220    5510 generic.go:356] "Generic (PLEG): container finished" podID="f48ed9e9-7396-4f40-8074-01195481a5af" containerID="76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1" exitCode=0
Nov 01 06:54:02.423353 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.423271    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv" event={"ID":"f48ed9e9-7396-4f40-8074-01195481a5af","Type":"ContainerDied","Data":"76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1"}
Nov 01 06:54:02.423353 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.423283    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"
Nov 01 06:54:02.423353 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.423304    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv" event={"ID":"f48ed9e9-7396-4f40-8074-01195481a5af","Type":"ContainerDied","Data":"9c807a8336dda31e1b7b79ec4dd1073a2a90b1146cf3a2510e059b502b73143a"}
Nov 01 06:54:02.430735 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.430718    5510 scope.go:122] "RemoveContainer" containerID="221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0"
Nov 01 06:54:02.431015 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:54:02.430990    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0\": container with ID starting with 221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0 not found: ID does not exist" containerID="221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0"
Nov 01 06:54:02.431063 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.431025    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0"} err="failed to get container status \"221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0\": rpc error: code = NotFound desc = could not find container \"221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0\": container with ID starting with 221b0a38b604e7d78fdc57946c086b6e1ab468e9ad10526c89d66ce40e4388d0 not found: ID does not exist"
Nov 01 06:54:02.431063 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.431048    5510 scope.go:122] "RemoveContainer" containerID="76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1"
Nov 01 06:54:02.439815 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.439797    5510 scope.go:122] "RemoveContainer" containerID="76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1"
Nov 01 06:54:02.440102 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:54:02.440085    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1\": container with ID starting with 76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1 not found: ID does not exist" containerID="76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1"
Nov 01 06:54:02.440162 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.440108    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1"} err="failed to get container status \"76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1\": rpc error: code = NotFound desc = could not find container \"76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1\": container with ID starting with 76fcd754e87a3791e0e17f2288f18ab5734a5f4cdfa6b447c43806f8aa2184c1 not found: ID does not exist"
Nov 01 06:54:02.584897 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.584871    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"]
Nov 01 06:54:02.633738 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.633703    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3986bf-89d5576c-zxgqv"]
Nov 01 06:54:02.696427 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.696351    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"]
Nov 01 06:54:02.700620 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:02.700595    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3986bf-655bdb7c49-nmmhp"]
Nov 01 06:54:03.235002 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:03.234965    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" path="/var/lib/kubelet/pods/c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8/volumes"
Nov 01 06:54:03.235510 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:03.235491    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48ed9e9-7396-4f40-8074-01195481a5af" path="/var/lib/kubelet/pods/f48ed9e9-7396-4f40-8074-01195481a5af/volumes"
Nov 01 06:54:04.451889 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:04.451854    5510 patch_prober.go:29] interesting pod/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:54:04.452262 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:04.451915    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:54:05.117678 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.117656    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:54:05.238909 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.238880    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds" podUID="4e157229-ba78-4611-91b8-270c29c0dcdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:54:05.286977 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.286946    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istiod-ca-cert\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.286993    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-kube-api-access-dmszf\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-kube-api-access-dmszf\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287040    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-token\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-token\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287086    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-credential-socket\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287156 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287136    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-data\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287200    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-envoy\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287276    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-socket\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287338    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-certs\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287387 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287368    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-podinfo\") pod \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\" (UID: \"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf\") "
Nov 01 06:54:05.287577 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287384    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-credential-socket" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:05.287577 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287453    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-data" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:05.287674 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287602    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-socket" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:05.288055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287727    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.288055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287751    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.288055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.287997    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-certs" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:05.288055 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.288030    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.290492 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.290011    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-token" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:05.290581 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.290562    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istiod-ca-cert" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:05.290853 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.290802    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-podinfo" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:54:05.291052 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.291028    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-envoy" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:05.291300 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.291277    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-kube-api-access-dmszf" pod "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" (UID: "7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf"). InnerVolumeSpecName "kube-api-access-dmszf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:05.388696 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.388667    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.388696 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.388692    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.388902 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.388701    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmszf\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-kube-api-access-dmszf\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.388902 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.388711    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.388902 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.388720    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.388902 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.388728    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:05.442371 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.442342    5510 generic.go:356] "Generic (PLEG): container finished" podID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerID="a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1" exitCode=0
Nov 01 06:54:05.442500 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.442386    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" event={"ID":"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf","Type":"ContainerDied","Data":"a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1"}
Nov 01 06:54:05.442500 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.442393    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"
Nov 01 06:54:05.442500 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.442427    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf" event={"ID":"7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf","Type":"ContainerDied","Data":"798bf0ec4d9f802e5cd3e6de0aa9a3b5f4bbdb07baa548c4d4ae763fe219258c"}
Nov 01 06:54:05.442500 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.442448    5510 scope.go:122] "RemoveContainer" containerID="a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1"
Nov 01 06:54:05.451851 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.451818    5510 scope.go:122] "RemoveContainer" containerID="3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886"
Nov 01 06:54:05.459622 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.459605    5510 scope.go:122] "RemoveContainer" containerID="a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1"
Nov 01 06:54:05.459957 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:54:05.459938    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1\": container with ID starting with a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1 not found: ID does not exist" containerID="a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1"
Nov 01 06:54:05.459995 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.459964    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1"} err="failed to get container status \"a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1\": rpc error: code = NotFound desc = could not find container \"a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1\": container with ID starting with a11912bfdf1bc23d6c36b37136ebea0c53b389171a516363e3ccc52c5d743df1 not found: ID does not exist"
Nov 01 06:54:05.459995 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.459981    5510 scope.go:122] "RemoveContainer" containerID="3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886"
Nov 01 06:54:05.460232 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:54:05.460217    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886\": container with ID starting with 3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886 not found: ID does not exist" containerID="3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886"
Nov 01 06:54:05.460282 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.460237    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886"} err="failed to get container status \"3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886\": rpc error: code = NotFound desc = could not find container \"3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886\": container with ID starting with 3477d85cd06c50db978f10ce9a168cd53f21e29ce06b16cefd2a9a334c992886 not found: ID does not exist"
Nov 01 06:54:05.606064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.605992    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"]
Nov 01 06:54:05.609497 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:05.609475    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3986bf-openshift-default-5f95f6d5c7-2hhtf"]
Nov 01 06:54:06.243665 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:06.243629    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m" podUID="357950be-1cc3-4216-95a9-376698aebd91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:54:06.801354 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:06.801318    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:54:07.235014 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:07.234982    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" path="/var/lib/kubelet/pods/7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf/volumes"
Nov 01 06:54:15.238383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.238347    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds" podUID="4e157229-ba78-4611-91b8-270c29c0dcdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:54:15.494193 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.494160    5510 generic.go:356] "Generic (PLEG): container finished" podID="4e157229-ba78-4611-91b8-270c29c0dcdc" containerID="84c5f81542394ffdf83fe73683db492ace80ca8db8cdeab51c16652ac8e332ea" exitCode=0
Nov 01 06:54:15.494342 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.494232    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds" event={"ID":"4e157229-ba78-4611-91b8-270c29c0dcdc","Type":"ContainerDied","Data":"84c5f81542394ffdf83fe73683db492ace80ca8db8cdeab51c16652ac8e332ea"}
Nov 01 06:54:15.495809 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.495789    5510 generic.go:356] "Generic (PLEG): container finished" podID="357950be-1cc3-4216-95a9-376698aebd91" containerID="2e34985c8f92d54b7d23fd784cbb2a588223e01b1001e96b3c41266fadca465a" exitCode=0
Nov 01 06:54:15.495969 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.495856    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m" event={"ID":"357950be-1cc3-4216-95a9-376698aebd91","Type":"ContainerDied","Data":"2e34985c8f92d54b7d23fd784cbb2a588223e01b1001e96b3c41266fadca465a"}
Nov 01 06:54:15.515684 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.515375    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 06:54:15.585732 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.585701    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357950be-1cc3-4216-95a9-376698aebd91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357950be-1cc3-4216-95a9-376698aebd91-plugins-config-volume\") pod \"357950be-1cc3-4216-95a9-376698aebd91\" (UID: \"357950be-1cc3-4216-95a9-376698aebd91\") "
Nov 01 06:54:15.585932 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.585818    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357950be-1cc3-4216-95a9-376698aebd91-kube-api-access-k27ft\" (UniqueName: \"kubernetes.io/projected/357950be-1cc3-4216-95a9-376698aebd91-kube-api-access-k27ft\") pod \"357950be-1cc3-4216-95a9-376698aebd91\" (UID: \"357950be-1cc3-4216-95a9-376698aebd91\") "
Nov 01 06:54:15.586133 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.586108    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357950be-1cc3-4216-95a9-376698aebd91-plugins-config-volume" pod "357950be-1cc3-4216-95a9-376698aebd91" (UID: "357950be-1cc3-4216-95a9-376698aebd91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:15.588041 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.588014    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357950be-1cc3-4216-95a9-376698aebd91-kube-api-access-k27ft" pod "357950be-1cc3-4216-95a9-376698aebd91" (UID: "357950be-1cc3-4216-95a9-376698aebd91"). InnerVolumeSpecName "kube-api-access-k27ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:15.603557 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.603537    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 06:54:15.686761 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.686733    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e157229-ba78-4611-91b8-270c29c0dcdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e157229-ba78-4611-91b8-270c29c0dcdc-plugins-config-volume\") pod \"4e157229-ba78-4611-91b8-270c29c0dcdc\" (UID: \"4e157229-ba78-4611-91b8-270c29c0dcdc\") "
Nov 01 06:54:15.686938 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.686783    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e157229-ba78-4611-91b8-270c29c0dcdc-kube-api-access-wxklq\" (UniqueName: \"kubernetes.io/projected/4e157229-ba78-4611-91b8-270c29c0dcdc-kube-api-access-wxklq\") pod \"4e157229-ba78-4611-91b8-270c29c0dcdc\" (UID: \"4e157229-ba78-4611-91b8-270c29c0dcdc\") "
Nov 01 06:54:15.687086 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.687074    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27ft\" (UniqueName: \"kubernetes.io/projected/357950be-1cc3-4216-95a9-376698aebd91-kube-api-access-k27ft\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:15.687137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.687089    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357950be-1cc3-4216-95a9-376698aebd91-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:15.687137 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.687119    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e157229-ba78-4611-91b8-270c29c0dcdc-plugins-config-volume" pod "4e157229-ba78-4611-91b8-270c29c0dcdc" (UID: "4e157229-ba78-4611-91b8-270c29c0dcdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:15.689026 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.689007    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e157229-ba78-4611-91b8-270c29c0dcdc-kube-api-access-wxklq" pod "4e157229-ba78-4611-91b8-270c29c0dcdc" (UID: "4e157229-ba78-4611-91b8-270c29c0dcdc"). InnerVolumeSpecName "kube-api-access-wxklq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:15.788509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.788435    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e157229-ba78-4611-91b8-270c29c0dcdc-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:15.788509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:15.788459    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxklq\" (UniqueName: \"kubernetes.io/projected/4e157229-ba78-4611-91b8-270c29c0dcdc-kube-api-access-wxklq\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:16.502116 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.502076    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"
Nov 01 06:54:16.502116 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.502102    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds" event={"ID":"4e157229-ba78-4611-91b8-270c29c0dcdc","Type":"ContainerDied","Data":"ed18e338121c9505597552793452e12c376a2ccb91700951f585265f4df540aa"}
Nov 01 06:54:16.502603 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.502170    5510 scope.go:122] "RemoveContainer" containerID="84c5f81542394ffdf83fe73683db492ace80ca8db8cdeab51c16652ac8e332ea"
Nov 01 06:54:16.503533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.503471    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m" event={"ID":"357950be-1cc3-4216-95a9-376698aebd91","Type":"ContainerDied","Data":"cf1bc22cc50a5601c61daaad83c39a73372afee7b90c61aacb521ee8d8adf0c7"}
Nov 01 06:54:16.503596 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.503531    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"
Nov 01 06:54:16.510907 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.510885    5510 scope.go:122] "RemoveContainer" containerID="2e34985c8f92d54b7d23fd784cbb2a588223e01b1001e96b3c41266fadca465a"
Nov 01 06:54:16.665173 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.665099    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"]
Nov 01 06:54:16.715035 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.715007    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9c8fc5-6c5d9b695-b82ds"]
Nov 01 06:54:16.778061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.778031    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"]
Nov 01 06:54:16.783951 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:16.783921    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9c8fc5-58b8d6ccdc-ftw2m"]
Nov 01 06:54:17.234488 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:17.234455    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357950be-1cc3-4216-95a9-376698aebd91" path="/var/lib/kubelet/pods/357950be-1cc3-4216-95a9-376698aebd91/volumes"
Nov 01 06:54:17.234803 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:17.234790    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e157229-ba78-4611-91b8-270c29c0dcdc" path="/var/lib/kubelet/pods/4e157229-ba78-4611-91b8-270c29c0dcdc/volumes"
Nov 01 06:54:17.544589 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:54:17.544506    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:54:30.568570 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:30.568498    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"]
Nov 01 06:54:30.569133 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:30.568822    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy" containerID="cri-o://a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e" gracePeriod=30
Nov 01 06:54:31.077233 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:31.077201    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:54:31.077407 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:31.077257    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:54:32.789543 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:32.789516    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l458f_7e89c29e-4c5c-4b3b-91bc-33014e337036/kuadrant-console-plugin/0.log"
Nov 01 06:54:35.813338 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.813315    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 06:54:35.874733 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.874659    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istiod-ca-cert\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.874733 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.874706    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-certs\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.874970 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.874748    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-token\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-token\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.874970 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.874769    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-socket\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.874970 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.874797    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-credential-socket\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.874970 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.874891    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-envoy\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.874970 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.874925    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-kube-api-access-gkd75\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-kube-api-access-gkd75\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.875243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875022    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-podinfo\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.875243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875059    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-data\") pod \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\" (UID: \"8efa1d66-775f-4ba4-ae2e-658da02e7ad0\") "
Nov 01 06:54:35.875243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875097    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-certs" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.875243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875120    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-socket" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.875243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875132    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istiod-ca-cert" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:35.875243 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875210    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-credential-socket" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.875536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875363    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.875536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875382    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.875536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875400    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.875536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875419    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.875536 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.875419    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-data" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.877187 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.877165    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-token" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:35.877293 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.877230    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-podinfo" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:54:35.877645 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.877623    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-envoy" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.877645 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.877635    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-kube-api-access-gkd75" pod "8efa1d66-775f-4ba4-ae2e-658da02e7ad0" (UID: "8efa1d66-775f-4ba4-ae2e-658da02e7ad0"). InnerVolumeSpecName "kube-api-access-gkd75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:35.976463 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.976430    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.976463 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.976461    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.976463 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.976471    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.976692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.976479    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.976692 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:35.976489    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkd75\" (UniqueName: \"kubernetes.io/projected/8efa1d66-775f-4ba4-ae2e-658da02e7ad0-kube-api-access-gkd75\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:36.601110 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.601074    5510 generic.go:356] "Generic (PLEG): container finished" podID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerID="a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e" exitCode=0
Nov 01 06:54:36.601284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.601125    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"
Nov 01 06:54:36.601284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.601126    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" event={"ID":"8efa1d66-775f-4ba4-ae2e-658da02e7ad0","Type":"ContainerDied","Data":"a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e"}
Nov 01 06:54:36.601284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.601246    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs" event={"ID":"8efa1d66-775f-4ba4-ae2e-658da02e7ad0","Type":"ContainerDied","Data":"a065c4a0e413551690144f87a52fdfa5f8e126a04b4c847f1dfd256ec64f8b64"}
Nov 01 06:54:36.601284 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.601274    5510 scope.go:122] "RemoveContainer" containerID="a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e"
Nov 01 06:54:36.610317 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.610295    5510 scope.go:122] "RemoveContainer" containerID="be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f"
Nov 01 06:54:36.619680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.619661    5510 scope.go:122] "RemoveContainer" containerID="a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e"
Nov 01 06:54:36.620055 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:54:36.620022    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e\": container with ID starting with a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e not found: ID does not exist" containerID="a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e"
Nov 01 06:54:36.620152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.620056    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e"} err="failed to get container status \"a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e\": rpc error: code = NotFound desc = could not find container \"a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e\": container with ID starting with a61b06e5b4859c62289d02db04268cef1cb79383a6e71d194390fda8778cb91e not found: ID does not exist"
Nov 01 06:54:36.620152 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.620082    5510 scope.go:122] "RemoveContainer" containerID="be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f"
Nov 01 06:54:36.620422 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:54:36.620404    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f\": container with ID starting with be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f not found: ID does not exist" containerID="be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f"
Nov 01 06:54:36.620489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.620434    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f"} err="failed to get container status \"be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f\": rpc error: code = NotFound desc = could not find container \"be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f\": container with ID starting with be976ec06edbdc620ee7a62319c52e8ff9706fbed5d8d3d073c6fef785c12c6f not found: ID does not exist"
Nov 01 06:54:36.813556 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.813523    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"]
Nov 01 06:54:36.863100 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:36.863025    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-sjgjs"]
Nov 01 06:54:37.234669 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:37.234639    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" path="/var/lib/kubelet/pods/8efa1d66-775f-4ba4-ae2e-658da02e7ad0/volumes"
Nov 01 06:54:44.361334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:44.361302    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f4wws_def3f09f-380f-4342-9f91-740df732a7fe/limitador/0.log"
Nov 01 06:54:45.896562 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:45.896529    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7684cd596b-68gdk_05da642c-8901-434c-859a-ad7386a0302c/manager/0.log"
Nov 01 06:54:46.553571 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:54:46.553540    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2vn9w_d2201b73-9fec-485b-ba93-ef6a56393bca/manager/0.log"
Nov 01 06:54:47.543677 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:54:47.543639    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:55:02.531734 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.531692    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"]
Nov 01 06:55:02.532273 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532256    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f48ed9e9-7396-4f40-8074-01195481a5af" containerName="payload-pre-processing"
Nov 01 06:55:02.532320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532275    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48ed9e9-7396-4f40-8074-01195481a5af" containerName="payload-pre-processing"
Nov 01 06:55:02.532320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532291    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy"
Nov 01 06:55:02.532320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532297    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy"
Nov 01 06:55:02.532320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532308    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy"
Nov 01 06:55:02.532320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532312    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy"
Nov 01 06:55:02.532320 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532320    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" containerName="main"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532325    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" containerName="main"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532334    5510 cpu_manager.go:414] "containerMap: removing container" podUID="aec10f38-d017-4b23-9974-6a5ee329c0b6" containerName="revoke-keys"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532338    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec10f38-d017-4b23-9974-6a5ee329c0b6" containerName="revoke-keys"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532347    5510 cpu_manager.go:414] "containerMap: removing container" podUID="357950be-1cc3-4216-95a9-376698aebd91" containerName="payload-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532352    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357950be-1cc3-4216-95a9-376698aebd91" containerName="payload-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532358    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ccd4246d-28af-4510-a7cd-d02e63d6fb8b" containerName="main"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532362    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd4246d-28af-4510-a7cd-d02e63d6fb8b" containerName="main"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532374    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532379    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532390    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532394    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532400    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" containerName="payload-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532407    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" containerName="payload-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532421    5510 cpu_manager.go:414] "containerMap: removing container" podUID="4e157229-ba78-4611-91b8-270c29c0dcdc" containerName="payload-pre-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532427    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e157229-ba78-4611-91b8-270c29c0dcdc" containerName="payload-pre-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532489    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e157229-ba78-4611-91b8-270c29c0dcdc" containerName="payload-pre-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532498    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48ed9e9-7396-4f40-8074-01195481a5af" containerName="payload-pre-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532504    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532510    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532515    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc62f94-c7cd-4b71-aaa4-ce3b32589ba9" containerName="main"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532522    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec10f38-d017-4b23-9974-6a5ee329c0b6" containerName="revoke-keys"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532528    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="357950be-1cc3-4216-95a9-376698aebd91" containerName="payload-processing"
Nov 01 06:55:02.532525 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532533    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd4246d-28af-4510-a7cd-d02e63d6fb8b" containerName="main"
Nov 01 06:55:02.533283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532539    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc671d8-f5c2-46c7-a55b-2cdea3bc98cf" containerName="istio-proxy"
Nov 01 06:55:02.533283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532545    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efa1d66-775f-4ba4-ae2e-658da02e7ad0" containerName="istio-proxy"
Nov 01 06:55:02.533283 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.532550    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d84a2e-e1e2-4107-ad0e-0e70d9175fc8" containerName="payload-processing"
Nov 01 06:55:02.535816 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.535795    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:02.587683 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.587654    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-main-045f69\""
Nov 01 06:55:02.617278 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.617246    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvz8h\" (UniqueName: \"kubernetes.io/projected/baa242a0-7d0f-423c-8278-cecdd18222e8-kube-api-access-nvz8h\") pod \"payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2\" (UID: \"baa242a0-7d0f-423c-8278-cecdd18222e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:02.617445 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.617303    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa242a0-7d0f-423c-8278-cecdd18222e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2\" (UID: \"baa242a0-7d0f-423c-8278-cecdd18222e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:02.638598 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.638560    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"]
Nov 01 06:55:02.717797 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.717761    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvz8h\" (UniqueName: \"kubernetes.io/projected/baa242a0-7d0f-423c-8278-cecdd18222e8-kube-api-access-nvz8h\") pod \"payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2\" (UID: \"baa242a0-7d0f-423c-8278-cecdd18222e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:02.717999 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.717815    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa242a0-7d0f-423c-8278-cecdd18222e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2\" (UID: \"baa242a0-7d0f-423c-8278-cecdd18222e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:02.718522 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.718502    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa242a0-7d0f-423c-8278-cecdd18222e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2\" (UID: \"baa242a0-7d0f-423c-8278-cecdd18222e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:02.778683 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.778649    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvz8h\" (UniqueName: \"kubernetes.io/projected/baa242a0-7d0f-423c-8278-cecdd18222e8-kube-api-access-nvz8h\") pod \"payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2\" (UID: \"baa242a0-7d0f-423c-8278-cecdd18222e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:02.849472 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:02.849388    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:03.232855 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:55:03.232561    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa242a0_7d0f_423c_8278_cecdd18222e8.slice/crio-7ed671aecd1b4dd0203937ae90e593b632f25c9181f93126ab74f60c92285a75 WatchSource:0}: Error finding container 7ed671aecd1b4dd0203937ae90e593b632f25c9181f93126ab74f60c92285a75: Status 404 returned error can't find the container with id 7ed671aecd1b4dd0203937ae90e593b632f25c9181f93126ab74f60c92285a75
Nov 01 06:55:03.238644 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:03.238614    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"]
Nov 01 06:55:03.731521 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:03.731481    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2" event={"ID":"baa242a0-7d0f-423c-8278-cecdd18222e8","Type":"ContainerStarted","Data":"4d446cfa7eb34cea2488506f7d46e5b9a3ecf09c638151aab70cfc938b9049ab"}
Nov 01 06:55:03.732002 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:03.731570    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2" event={"ID":"baa242a0-7d0f-423c-8278-cecdd18222e8","Type":"ContainerStarted","Data":"7ed671aecd1b4dd0203937ae90e593b632f25c9181f93126ab74f60c92285a75"}
Nov 01 06:55:03.732002 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:03.731635    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:03.846794 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:03.846736    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2" podStartSLOduration=1.846720812 podStartE2EDuration="1.846720812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:03.84519897 +0000 UTC m=+4140.723172379" watchObservedRunningTime="2026-11-01 06:55:03.846720812 +0000 UTC m=+4140.724694216"
Nov 01 06:55:09.151267 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:09.151223    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"]
Nov 01 06:55:09.151789 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:09.151564    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy" containerID="cri-o://d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec" gracePeriod=30
Nov 01 06:55:12.973379 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:12.973342    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:55:14.400112 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.400083    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:55:14.429334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429247    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-credential-socket\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.429334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429307    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-podinfo\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.429546 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429344    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66a3206-54e5-419a-a623-ba67e69b3b6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66a3206-54e5-419a-a623-ba67e69b3b6b-istiod-ca-cert\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.429546 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429376    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-data\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.429546 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429421    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-token\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-token\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.429546 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429456    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-kube-api-access-kwkc8\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-kube-api-access-kwkc8\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.429546 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429506    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-envoy\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.429812 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429576    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-credential-socket" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:14.429812 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429687    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-data" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:14.429812 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429771    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66a3206-54e5-419a-a623-ba67e69b3b6b-istiod-ca-cert" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:55:14.430028 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.429977    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-socket\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.430092 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.430030    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-certs\") pod \"e66a3206-54e5-419a-a623-ba67e69b3b6b\" (UID: \"e66a3206-54e5-419a-a623-ba67e69b3b6b\") "
Nov 01 06:55:14.430332 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.430314    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.430435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.430376    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66a3206-54e5-419a-a623-ba67e69b3b6b-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.430435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.430392    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.430435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.430409    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-socket" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:14.430538 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.430479    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-certs" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:14.431884 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.431859    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-podinfo" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:55:14.431989 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.431901    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-kube-api-access-kwkc8" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "kube-api-access-kwkc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:55:14.432247 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.432220    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-envoy" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:14.432301 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.432236    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-token" pod "e66a3206-54e5-419a-a623-ba67e69b3b6b" (UID: "e66a3206-54e5-419a-a623-ba67e69b3b6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:55:14.531236 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.531198    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.531236 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.531227    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwkc8\" (UniqueName: \"kubernetes.io/projected/e66a3206-54e5-419a-a623-ba67e69b3b6b-kube-api-access-kwkc8\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.531236 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.531241    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.531489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.531250    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.531489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.531258    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66a3206-54e5-419a-a623-ba67e69b3b6b-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.531489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.531267    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66a3206-54e5-419a-a623-ba67e69b3b6b-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:14.738973 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.738945    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 06:55:14.781136 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.781106    5510 generic.go:356] "Generic (PLEG): container finished" podID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerID="d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec" exitCode=0
Nov 01 06:55:14.781327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.781183    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"
Nov 01 06:55:14.781327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.781192    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" event={"ID":"e66a3206-54e5-419a-a623-ba67e69b3b6b","Type":"ContainerDied","Data":"d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec"}
Nov 01 06:55:14.781327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.781243    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l" event={"ID":"e66a3206-54e5-419a-a623-ba67e69b3b6b","Type":"ContainerDied","Data":"a49ec111685c9f4c9b0b9e3b11476a29ea1716146b88800533f89f5ab2475582"}
Nov 01 06:55:14.781327 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.781275    5510 scope.go:122] "RemoveContainer" containerID="d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec"
Nov 01 06:55:14.791622 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.791604    5510 scope.go:122] "RemoveContainer" containerID="d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec"
Nov 01 06:55:14.791955 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:55:14.791936    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec\": container with ID starting with d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec not found: ID does not exist" containerID="d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec"
Nov 01 06:55:14.792028 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.791963    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec"} err="failed to get container status \"d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec\": rpc error: code = NotFound desc = could not find container \"d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec\": container with ID starting with d267876a284259a28a94fc7406e3264b2e1ae980ab46a082b5f200483dc687ec not found: ID does not exist"
Nov 01 06:55:14.966466 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.966406    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"]
Nov 01 06:55:14.968143 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:14.968113    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-xgr5l"]
Nov 01 06:55:15.234405 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:15.234373    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" path="/var/lib/kubelet/pods/e66a3206-54e5-419a-a623-ba67e69b3b6b/volumes"
Nov 01 06:55:17.546360 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:55:17.546327    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:55:32.407314 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:32.407281    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:55:47.544709 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:55:47.544669    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:55:59.696842 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:55:59.696804    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:56:17.544704 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:56:17.544667    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:56:24.870608 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:56:24.870573    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:56:47.546213 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:56:47.546131    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:57:06.388433 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.388392    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"]
Nov 01 06:57:06.389036 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.389014    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy"
Nov 01 06:57:06.389119 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.389038    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy"
Nov 01 06:57:06.389180 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.389159    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66a3206-54e5-419a-a623-ba67e69b3b6b" containerName="istio-proxy"
Nov 01 06:57:06.392341 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.392321    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.444432 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.444407    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:57:06.493711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.493674    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:57:06.493711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.493695    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:57:06.493711 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.493680    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:57:06.495093 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.495076    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"]
Nov 01 06:57:06.531642 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.531620    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.531762 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.531672    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngsc\" (UniqueName: \"kubernetes.io/projected/a971632d-0719-4b78-b684-6ce7e5931532-kube-api-access-wngsc\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.531762 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.531748    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.632357 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.632321    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.632513 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.632390    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngsc\" (UniqueName: \"kubernetes.io/projected/a971632d-0719-4b78-b684-6ce7e5931532-kube-api-access-wngsc\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.632513 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.632423    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.633118 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.633094    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.634751 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.634732    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.693313 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.693246    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngsc\" (UniqueName: \"kubernetes.io/projected/a971632d-0719-4b78-b684-6ce7e5931532-kube-api-access-wngsc\") pod \"maas-api-revoke-sub-85237e45-95cqs\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:06.705028 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:06.705004    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:07.073958 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:07.073926    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"]
Nov 01 06:57:07.076661 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:57:07.076635    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda971632d_0719_4b78_b684_6ce7e5931532.slice/crio-fdc29e3001552c34695b23fb24881f985e7c260911cfdbefec3e9d533c982162 WatchSource:0}: Error finding container fdc29e3001552c34695b23fb24881f985e7c260911cfdbefec3e9d533c982162: Status 404 returned error can't find the container with id fdc29e3001552c34695b23fb24881f985e7c260911cfdbefec3e9d533c982162
Nov 01 06:57:07.299539 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:07.299514    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs" event={"ID":"a971632d-0719-4b78-b684-6ce7e5931532","Type":"ContainerStarted","Data":"fdc29e3001552c34695b23fb24881f985e7c260911cfdbefec3e9d533c982162"}
Nov 01 06:57:08.305696 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:08.305607    5510 generic.go:356] "Generic (PLEG): container finished" podID="a971632d-0719-4b78-b684-6ce7e5931532" containerID="b55169b407d538dcde483a1ac88398ff95fbe75fe357d001ed2c509b57e4051f" exitCode=0
Nov 01 06:57:08.306134 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:08.305695    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs" event={"ID":"a971632d-0719-4b78-b684-6ce7e5931532","Type":"ContainerDied","Data":"b55169b407d538dcde483a1ac88398ff95fbe75fe357d001ed2c509b57e4051f"}
Nov 01 06:57:09.187511 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.187478    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"]
Nov 01 06:57:09.191098 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.191080    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257302 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257273    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257469 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257320    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257469 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257345    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257469 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257415    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d488d9-60bc-480d-b851-cea9d9a4edb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257469 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257467    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257621 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257496    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257621 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257515    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257621 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257572    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.257717 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.257652    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfn6\" (UniqueName: \"kubernetes.io/projected/47d488d9-60bc-480d-b851-cea9d9a4edb7-kube-api-access-8lfn6\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.299346 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.299314    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"]
Nov 01 06:57:09.358811 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.358773    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.358890    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.358922    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359033    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d488d9-60bc-480d-b851-cea9d9a4edb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359078    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359132    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359163    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359255 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359212    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359291    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfn6\" (UniqueName: \"kubernetes.io/projected/47d488d9-60bc-480d-b851-cea9d9a4edb7-kube-api-access-8lfn6\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359306    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359211    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359589    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359641 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359626    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.359883 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.359863    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d488d9-60bc-480d-b851-cea9d9a4edb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.361778 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.361750    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.362083 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.362067    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.419984 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.419957    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfn6\" (UniqueName: \"kubernetes.io/projected/47d488d9-60bc-480d-b851-cea9d9a4edb7-kube-api-access-8lfn6\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.420118 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.419966    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d488d9-60bc-480d-b851-cea9d9a4edb7-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-f9ttw\" (UID: \"47d488d9-60bc-480d-b851-cea9d9a4edb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.444730 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.444677    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:09.503735 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.503711    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:09.561564 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.561525    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-xks-ca\") pod \"a971632d-0719-4b78-b684-6ce7e5931532\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") "
Nov 01 06:57:09.561717 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.561609    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-openshift-ca\") pod \"a971632d-0719-4b78-b684-6ce7e5931532\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") "
Nov 01 06:57:09.561717 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.561696    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a971632d-0719-4b78-b684-6ce7e5931532-kube-api-access-wngsc\" (UniqueName: \"kubernetes.io/projected/a971632d-0719-4b78-b684-6ce7e5931532-kube-api-access-wngsc\") pod \"a971632d-0719-4b78-b684-6ce7e5931532\" (UID: \"a971632d-0719-4b78-b684-6ce7e5931532\") "
Nov 01 06:57:09.563101 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.563072    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-openshift-ca" pod "a971632d-0719-4b78-b684-6ce7e5931532" (UID: "a971632d-0719-4b78-b684-6ce7e5931532"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:57:09.569037 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.568201    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a971632d-0719-4b78-b684-6ce7e5931532-kube-api-access-wngsc" pod "a971632d-0719-4b78-b684-6ce7e5931532" (UID: "a971632d-0719-4b78-b684-6ce7e5931532"). InnerVolumeSpecName "kube-api-access-wngsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:57:09.570662 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.570642    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-xks-ca" pod "a971632d-0719-4b78-b684-6ce7e5931532" (UID: "a971632d-0719-4b78-b684-6ce7e5931532"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:57:09.663420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.663387    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngsc\" (UniqueName: \"kubernetes.io/projected/a971632d-0719-4b78-b684-6ce7e5931532-kube-api-access-wngsc\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:57:09.663420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.663416    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:57:09.663420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.663425    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971632d-0719-4b78-b684-6ce7e5931532-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:57:09.926864 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.926819    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"]
Nov 01 06:57:09.928958 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:57:09.928930    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d488d9_60bc_480d_b851_cea9d9a4edb7.slice/crio-0d196c176bd1f726dc3fc6d76de59ab79a210d55dd932e2a69f2129a26b15d5e WatchSource:0}: Error finding container 0d196c176bd1f726dc3fc6d76de59ab79a210d55dd932e2a69f2129a26b15d5e: Status 404 returned error can't find the container with id 0d196c176bd1f726dc3fc6d76de59ab79a210d55dd932e2a69f2129a26b15d5e
Nov 01 06:57:09.931127 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.931092    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:57:09.931216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.931160    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:57:09.931216 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:09.931197    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:57:10.317378 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.317339    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" event={"ID":"47d488d9-60bc-480d-b851-cea9d9a4edb7","Type":"ContainerStarted","Data":"db3e94ded5ff22e8e8201dd75a1e298abb1fd4db2153624a4be377d43f98c243"}
Nov 01 06:57:10.317956 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.317925    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" event={"ID":"47d488d9-60bc-480d-b851-cea9d9a4edb7","Type":"ContainerStarted","Data":"0d196c176bd1f726dc3fc6d76de59ab79a210d55dd932e2a69f2129a26b15d5e"}
Nov 01 06:57:10.319089 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.319018    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs" event={"ID":"a971632d-0719-4b78-b684-6ce7e5931532","Type":"ContainerDied","Data":"fdc29e3001552c34695b23fb24881f985e7c260911cfdbefec3e9d533c982162"}
Nov 01 06:57:10.319089 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.319066    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc29e3001552c34695b23fb24881f985e7c260911cfdbefec3e9d533c982162"
Nov 01 06:57:10.319089 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.319082    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"
Nov 01 06:57:10.438639 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.438582    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podStartSLOduration=1.4385643049999999 podStartE2EDuration="1.438564305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:10.436689583 +0000 UTC m=+4267.314663000" watchObservedRunningTime="2026-11-01 06:57:10.438564305 +0000 UTC m=+4267.316537718"
Nov 01 06:57:10.504461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.504431    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:10.505874 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.505850    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:57:10.505993 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:10.505902    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 01 06:57:11.504680 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:11.504649    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:57:11.505110 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:11.504709    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 01 06:57:12.505103 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:12.505069    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:57:12.505504 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:12.505132    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 01 06:57:13.504310 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:13.504267    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:57:13.504489 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:13.504335    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 01 06:57:14.504427 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:14.504383    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:57:14.504858 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:14.504444    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 01 06:57:15.504978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:15.504939    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:57:15.505383 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:15.505009    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 01 06:57:16.504476 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:16.504442    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:57:16.504653 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:16.504563    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 01 06:57:17.505177 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:17.505142    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:57:17.505585 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:17.505209    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 01 06:57:17.544902 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:57:17.544867    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:57:19.545978 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:19.545933    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:57:19.546509 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:19.545992    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:57:20.586961 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:20.586926    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:57:20.587392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:20.586979    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:57:20.590433 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:20.590413    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:20.590771 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:20.590754    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:20.703191 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:20.703160    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw"
Nov 01 06:57:24.178501 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.178465    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"]
Nov 01 06:57:24.178957 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.178859    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a971632d-0719-4b78-b684-6ce7e5931532" containerName="revoke-keys"
Nov 01 06:57:24.178957 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.178870    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a971632d-0719-4b78-b684-6ce7e5931532" containerName="revoke-keys"
Nov 01 06:57:24.178957 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.178951    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a971632d-0719-4b78-b684-6ce7e5931532" containerName="revoke-keys"
Nov 01 06:57:24.182023 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.182002    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.217933 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.217909    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.218039 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.217976    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.218092 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.218066    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.218148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.218093    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d6bee8-e720-4cda-bed6-aeba98f0b113-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.218148 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.218117    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.218274 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.218254    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.218334 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.218320    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.218392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.218348    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.218392 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.218375    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mrgq\" (UniqueName: \"kubernetes.io/projected/25d6bee8-e720-4cda-bed6-aeba98f0b113-kube-api-access-4mrgq\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319023 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.318993    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319038    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319065    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d6bee8-e720-4cda-bed6-aeba98f0b113-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319093    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319198 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319163    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319229    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319253    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319281    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mrgq\" (UniqueName: \"kubernetes.io/projected/25d6bee8-e720-4cda-bed6-aeba98f0b113-kube-api-access-4mrgq\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319420 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319343    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319649 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319455    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319649 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319539    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319760 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319646    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.319841 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.319801    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.320218 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.320199    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d6bee8-e720-4cda-bed6-aeba98f0b113-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.321626 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.321600    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.321780 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.321763    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.380240 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.380208    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mrgq\" (UniqueName: \"kubernetes.io/projected/25d6bee8-e720-4cda-bed6-aeba98f0b113-kube-api-access-4mrgq\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.380240 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.380229    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d6bee8-e720-4cda-bed6-aeba98f0b113-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-dtgh2\" (UID: \"25d6bee8-e720-4cda-bed6-aeba98f0b113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.385758 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.385732    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"]
Nov 01 06:57:24.494322 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.494293    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:24.919095 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.919072    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"]
Nov 01 06:57:24.921173 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:57:24.921148    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d6bee8_e720_4cda_bed6_aeba98f0b113.slice/crio-50033ef00eb4cca84708c5396e04fb4a2ebf395910a405bc279662660fc321cf WatchSource:0}: Error finding container 50033ef00eb4cca84708c5396e04fb4a2ebf395910a405bc279662660fc321cf: Status 404 returned error can't find the container with id 50033ef00eb4cca84708c5396e04fb4a2ebf395910a405bc279662660fc321cf
Nov 01 06:57:24.923530 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.923472    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:57:24.923616 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.923540    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:57:24.923616 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:24.923568    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:57:25.410279 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:25.410238    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" event={"ID":"25d6bee8-e720-4cda-bed6-aeba98f0b113","Type":"ContainerStarted","Data":"88a2caf3fdc98af0786df461dedb1fca9df687d59749e1a712805355e78799ee"}
Nov 01 06:57:25.410279 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:25.410289    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" event={"ID":"25d6bee8-e720-4cda-bed6-aeba98f0b113","Type":"ContainerStarted","Data":"50033ef00eb4cca84708c5396e04fb4a2ebf395910a405bc279662660fc321cf"}
Nov 01 06:57:25.495106 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:25.495066    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:25.496431 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:25.496400    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 06:57:25.496554 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:25.496458    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 06:57:25.530468 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:25.530412    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podStartSLOduration=1.530397771 podStartE2EDuration="1.530397771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:25.529495431 +0000 UTC m=+4282.407468845" watchObservedRunningTime="2026-11-01 06:57:25.530397771 +0000 UTC m=+4282.408371182"
Nov 01 06:57:26.495435 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:26.495397    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 06:57:26.495942 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:26.495452    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 06:57:27.495450 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:27.495407    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 06:57:27.495886 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:27.495486    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 06:57:28.495295 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:28.495263    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 06:57:28.495513 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:28.495312    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 06:57:29.494723 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:29.494680    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 06:57:29.494949 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:29.494751    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 06:57:30.495298 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:30.495262    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 06:57:30.495725 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:30.495312    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 06:57:31.495062 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:31.495026    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 06:57:31.495265 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:31.495083    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 06:57:32.495009 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:32.494965    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 06:57:32.495484 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:32.495031    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 06:57:33.512818 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:33.512767    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:33.514903 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:33.514878    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:33.515597 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:57:33.515579    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2"
Nov 01 06:57:47.544777 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:57:47.544737    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:58:17.544169 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:58:17.544087    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:58:41.044244 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:58:41.044207    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"]
Nov 01 06:58:41.093533 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:58:41.093504    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f54b64a-v56kg"]
Nov 01 06:58:41.234520 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:58:41.234490    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78bb2b36-be1a-4503-9a20-af256f5c6dd9" path="/var/lib/kubelet/pods/78bb2b36-be1a-4503-9a20-af256f5c6dd9/volumes"
Nov 01 06:58:47.544649 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:58:47.544609    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:58:51.045417 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:58:51.045389    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"]
Nov 01 06:58:51.094597 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:58:51.094568    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e876bf-zxc5c"]
Nov 01 06:58:51.234786 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:58:51.234754    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec10f38-d017-4b23-9974-6a5ee329c0b6" path="/var/lib/kubelet/pods/aec10f38-d017-4b23-9974-6a5ee329c0b6/volumes"
Nov 01 06:59:17.544589 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:59:17.544552    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:59:47.544005 ip-10-0-138-190 kubenswrapper[5510]: E1101 06:59:47.543926    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 06:59:52.888061 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:52.888026    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 06:59:56.969461 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:56.969426    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"]
Nov 01 06:59:56.973064 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:56.973046    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.025065 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.025043    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:59:57.035595 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.035565    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.035724 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.035609    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-kube-api-access-tfmmf\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.035796 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.035770    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.044558 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.044535    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:59:57.044670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.044587    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:59:57.074496 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.074472    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 06:59:57.076075 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.076051    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"]
Nov 01 06:59:57.136670 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.136633    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-kube-api-access-tfmmf\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.136915 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.136715    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.136915 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.136749    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.137474 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.137451    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.139190 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.139171    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.197395 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.197360    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-kube-api-access-tfmmf\") pod \"maas-api-revoke-sub-d576e68b-hclkd\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.285217 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.285144    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 06:59:57.655331 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:57.655306    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"]
Nov 01 06:59:57.658129 ip-10-0-138-190 kubenswrapper[5510]: W1101 06:59:57.658103    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1805529_9dbf_4fe8_926e_395ce2e8fbfa.slice/crio-c05f39f6e21fef0bb6e3326d0c2af49bf30e9962b9ed6094d570cfdfb00ceca8 WatchSource:0}: Error finding container c05f39f6e21fef0bb6e3326d0c2af49bf30e9962b9ed6094d570cfdfb00ceca8: Status 404 returned error can't find the container with id c05f39f6e21fef0bb6e3326d0c2af49bf30e9962b9ed6094d570cfdfb00ceca8
Nov 01 06:59:58.156277 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:58.156239    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd" event={"ID":"c1805529-9dbf-4fe8-926e-395ce2e8fbfa","Type":"ContainerStarted","Data":"0fd159795c894e2d06761a68e081c631c19227108b6e8f527afcf8ab1e448246"}
Nov 01 06:59:58.156684 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:58.156297    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd" event={"ID":"c1805529-9dbf-4fe8-926e-395ce2e8fbfa","Type":"ContainerStarted","Data":"c05f39f6e21fef0bb6e3326d0c2af49bf30e9962b9ed6094d570cfdfb00ceca8"}
Nov 01 06:59:58.271727 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:58.271664    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd" podStartSLOduration=2.271645418 podStartE2EDuration="2.271645418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:59:58.270259429 +0000 UTC m=+4435.148232838" watchObservedRunningTime="2026-11-01 06:59:58.271645418 +0000 UTC m=+4435.149618829"
Nov 01 06:59:59.162224 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:59.162185    5510 generic.go:356] "Generic (PLEG): container finished" podID="c1805529-9dbf-4fe8-926e-395ce2e8fbfa" containerID="0fd159795c894e2d06761a68e081c631c19227108b6e8f527afcf8ab1e448246" exitCode=0
Nov 01 06:59:59.162600 ip-10-0-138-190 kubenswrapper[5510]: I1101 06:59:59.162273    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd" event={"ID":"c1805529-9dbf-4fe8-926e-395ce2e8fbfa","Type":"ContainerDied","Data":"0fd159795c894e2d06761a68e081c631c19227108b6e8f527afcf8ab1e448246"}
Nov 01 07:00:00.168131 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.168102    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"]
Nov 01 07:00:00.171511 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.171494    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:00.231475 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.231446    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"]
Nov 01 07:00:00.265539 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.265504    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7749\" (UniqueName: \"kubernetes.io/projected/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-kube-api-access-q7749\") pod \"maas-api-key-cleanup-29891940-44p8j\" (UID: \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:00.265698 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.265622    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891940-44p8j\" (UID: \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:00.304163 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.304136    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 07:00:00.366447 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.366422    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-xks-ca\") pod \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") "
Nov 01 07:00:00.366620 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.366482    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-openshift-ca\") pod \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") "
Nov 01 07:00:00.366620 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.366520    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-kube-api-access-tfmmf\") pod \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\" (UID: \"c1805529-9dbf-4fe8-926e-395ce2e8fbfa\") "
Nov 01 07:00:00.366757 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.366649    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891940-44p8j\" (UID: \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:00.366757 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.366739    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7749\" (UniqueName: \"kubernetes.io/projected/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-kube-api-access-q7749\") pod \"maas-api-key-cleanup-29891940-44p8j\" (UID: \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:00.366971 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.366949    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-openshift-ca" pod "c1805529-9dbf-4fe8-926e-395ce2e8fbfa" (UID: "c1805529-9dbf-4fe8-926e-395ce2e8fbfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:00.367374 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.367357    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891940-44p8j\" (UID: \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:00.369271 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.369250    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-xks-ca" pod "c1805529-9dbf-4fe8-926e-395ce2e8fbfa" (UID: "c1805529-9dbf-4fe8-926e-395ce2e8fbfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:00:00.369359 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.369337    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-kube-api-access-tfmmf" pod "c1805529-9dbf-4fe8-926e-395ce2e8fbfa" (UID: "c1805529-9dbf-4fe8-926e-395ce2e8fbfa"). InnerVolumeSpecName "kube-api-access-tfmmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:00.427413 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.427353    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7749\" (UniqueName: \"kubernetes.io/projected/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-kube-api-access-q7749\") pod \"maas-api-key-cleanup-29891940-44p8j\" (UID: \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:00.467578 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.467554    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:00.467578 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.467577    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-kube-api-access-tfmmf\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:00.467729 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.467587    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1805529-9dbf-4fe8-926e-395ce2e8fbfa-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:00.485043 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.485027    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:00.854507 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:00.854484    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"]
Nov 01 07:00:00.856987 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:00:00.856963    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9cdf52_fea3_41cf_aa9c_d6098b176d38.slice/crio-0410a1071714ed130fe65f2b01d64ee1cc426e34a2283f24d5166d92120b9f1d WatchSource:0}: Error finding container 0410a1071714ed130fe65f2b01d64ee1cc426e34a2283f24d5166d92120b9f1d: Status 404 returned error can't find the container with id 0410a1071714ed130fe65f2b01d64ee1cc426e34a2283f24d5166d92120b9f1d
Nov 01 07:00:01.174839 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:01.174746    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j" event={"ID":"0b9cdf52-fea3-41cf-aa9c-d6098b176d38","Type":"ContainerStarted","Data":"0410a1071714ed130fe65f2b01d64ee1cc426e34a2283f24d5166d92120b9f1d"}
Nov 01 07:00:01.176085 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:01.176060    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd" event={"ID":"c1805529-9dbf-4fe8-926e-395ce2e8fbfa","Type":"ContainerDied","Data":"c05f39f6e21fef0bb6e3326d0c2af49bf30e9962b9ed6094d570cfdfb00ceca8"}
Nov 01 07:00:01.176216 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:01.176089    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"
Nov 01 07:00:01.176216 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:01.176098    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05f39f6e21fef0bb6e3326d0c2af49bf30e9962b9ed6094d570cfdfb00ceca8"
Nov 01 07:00:02.182735 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:02.182652    5510 generic.go:356] "Generic (PLEG): container finished" podID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerID="b03a7a69ac16413a981d0b3b768ec2837d68006c2900d817b952c3939bcac2cb" exitCode=60
Nov 01 07:00:02.182735 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:02.182722    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j" event={"ID":"0b9cdf52-fea3-41cf-aa9c-d6098b176d38","Type":"ContainerDied","Data":"b03a7a69ac16413a981d0b3b768ec2837d68006c2900d817b952c3939bcac2cb"}
Nov 01 07:00:02.183173 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:02.182988    5510 scope.go:122] "RemoveContainer" containerID="b03a7a69ac16413a981d0b3b768ec2837d68006c2900d817b952c3939bcac2cb"
Nov 01 07:00:03.189798 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:03.189760    5510 generic.go:356] "Generic (PLEG): container finished" podID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerID="4e88e099e9c3baf9a58971d76bd64371507280bc30d00176e03b394d3ef6cb02" exitCode=60
Nov 01 07:00:03.190319 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:03.189869    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j" event={"ID":"0b9cdf52-fea3-41cf-aa9c-d6098b176d38","Type":"ContainerDied","Data":"4e88e099e9c3baf9a58971d76bd64371507280bc30d00176e03b394d3ef6cb02"}
Nov 01 07:00:03.190319 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:03.189934    5510 scope.go:122] "RemoveContainer" containerID="b03a7a69ac16413a981d0b3b768ec2837d68006c2900d817b952c3939bcac2cb"
Nov 01 07:00:03.190319 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:03.190173    5510 scope.go:122] "RemoveContainer" containerID="4e88e099e9c3baf9a58971d76bd64371507280bc30d00176e03b394d3ef6cb02"
Nov 01 07:00:03.190476 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:00:03.190443    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891940-44p8j_odh-ai-gateway-infra(0b9cdf52-fea3-41cf-aa9c-d6098b176d38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38"
Nov 01 07:00:04.196936 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:04.196906    5510 scope.go:122] "RemoveContainer" containerID="4e88e099e9c3baf9a58971d76bd64371507280bc30d00176e03b394d3ef6cb02"
Nov 01 07:00:04.197321 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:00:04.197149    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891940-44p8j_odh-ai-gateway-infra(0b9cdf52-fea3-41cf-aa9c-d6098b176d38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38"
Nov 01 07:00:04.753680 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:04.753651    5510 scope.go:122] "RemoveContainer" containerID="a8511a15a3bdf8f6a5b4ad55de7737ffa7cf2aac47414c5b5460f4603c3a4aa9"
Nov 01 07:00:04.762653 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:04.762634    5510 scope.go:122] "RemoveContainer" containerID="5a768ff31b104ba1b98a7a68588f8cceb9bfe5cc5edc7c3dce240f1599484bd0"
Nov 01 07:00:11.901651 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:11.901613    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:00:16.228927 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:16.228893    5510 scope.go:122] "RemoveContainer" containerID="4e88e099e9c3baf9a58971d76bd64371507280bc30d00176e03b394d3ef6cb02"
Nov 01 07:00:17.256376 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:17.256287    5510 generic.go:356] "Generic (PLEG): container finished" podID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerID="b9dbb532e1dd70433687cf0a948d6324f1e19c18d001f142662b94ebec80afc5" exitCode=60
Nov 01 07:00:17.256792 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:17.256367    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j" event={"ID":"0b9cdf52-fea3-41cf-aa9c-d6098b176d38","Type":"ContainerDied","Data":"b9dbb532e1dd70433687cf0a948d6324f1e19c18d001f142662b94ebec80afc5"}
Nov 01 07:00:17.256792 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:17.256426    5510 scope.go:122] "RemoveContainer" containerID="4e88e099e9c3baf9a58971d76bd64371507280bc30d00176e03b394d3ef6cb02"
Nov 01 07:00:17.256792 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:17.256743    5510 scope.go:122] "RemoveContainer" containerID="b9dbb532e1dd70433687cf0a948d6324f1e19c18d001f142662b94ebec80afc5"
Nov 01 07:00:17.257032 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:00:17.257010    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891940-44p8j_odh-ai-gateway-infra(0b9cdf52-fea3-41cf-aa9c-d6098b176d38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38"
Nov 01 07:00:17.545012 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:00:17.544930    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:00:18.380605 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:18.380569    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"]
Nov 01 07:00:18.530082 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:18.530052    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:18.646783 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:18.646698    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-maas-api-service-ca\") pod \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\" (UID: \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\") "
Nov 01 07:00:18.647015 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:18.646803    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-kube-api-access-q7749\" (UniqueName: \"kubernetes.io/projected/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-kube-api-access-q7749\") pod \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\" (UID: \"0b9cdf52-fea3-41cf-aa9c-d6098b176d38\") "
Nov 01 07:00:18.647149 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:18.647125    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-maas-api-service-ca" pod "0b9cdf52-fea3-41cf-aa9c-d6098b176d38" (UID: "0b9cdf52-fea3-41cf-aa9c-d6098b176d38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:18.649532 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:18.649500    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-kube-api-access-q7749" pod "0b9cdf52-fea3-41cf-aa9c-d6098b176d38" (UID: "0b9cdf52-fea3-41cf-aa9c-d6098b176d38"). InnerVolumeSpecName "kube-api-access-q7749". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:18.748260 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:18.748214    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:18.748260 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:18.748261    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7749\" (UniqueName: \"kubernetes.io/projected/0b9cdf52-fea3-41cf-aa9c-d6098b176d38-kube-api-access-q7749\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:19.268679 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:19.268652    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"
Nov 01 07:00:19.268901 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:19.268647    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j" event={"ID":"0b9cdf52-fea3-41cf-aa9c-d6098b176d38","Type":"ContainerDied","Data":"0410a1071714ed130fe65f2b01d64ee1cc426e34a2283f24d5166d92120b9f1d"}
Nov 01 07:00:19.268901 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:19.268789    5510 scope.go:122] "RemoveContainer" containerID="b9dbb532e1dd70433687cf0a948d6324f1e19c18d001f142662b94ebec80afc5"
Nov 01 07:00:19.430642 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:19.430606    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"]
Nov 01 07:00:19.479742 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:19.479709    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-44p8j"]
Nov 01 07:00:21.243122 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:21.243090    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" path="/var/lib/kubelet/pods/0b9cdf52-fea3-41cf-aa9c-d6098b176d38/volumes"
Nov 01 07:00:21.634038 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:21.633940    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:00:21.634333 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:21.634302    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:00:24.192953 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.192916    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"]
Nov 01 07:00:24.193379 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193329    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.193379 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193340    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.193379 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193355    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.193379 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193360    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.193379 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193366    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.193379 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193371    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.193575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193405    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c1805529-9dbf-4fe8-926e-395ce2e8fbfa" containerName="revoke-keys"
Nov 01 07:00:24.193575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193410    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1805529-9dbf-4fe8-926e-395ce2e8fbfa" containerName="revoke-keys"
Nov 01 07:00:24.193575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193463    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.193575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193469    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.193575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193481    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1805529-9dbf-4fe8-926e-395ce2e8fbfa" containerName="revoke-keys"
Nov 01 07:00:24.193575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.193488    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9cdf52-fea3-41cf-aa9c-d6098b176d38" containerName="cleanup"
Nov 01 07:00:24.198140 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.198121    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.344868 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.344814    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wclfx\" (UniqueName: \"kubernetes.io/projected/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-kube-api-access-wclfx\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.345031 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.344943    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.345031 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.344995    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.345031 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.345014    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.345144 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.345031    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.345182 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.345146    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.345215 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.345190    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.345259 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.345240    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.345296 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.345268    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.401650 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.401619    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"]
Nov 01 07:00:24.446451 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446363    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.446451 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446417    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.446451 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446442    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.446738 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446466    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.446738 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446507    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.446738 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446553    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.446738 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446581    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.446738 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446603    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.446738 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446682    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wclfx\" (UniqueName: \"kubernetes.io/projected/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-kube-api-access-wclfx\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.447089 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446815    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.447089 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.446978    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.447089 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.447025    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.447248 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.447231    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.447382 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.447364    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.449484 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.449457    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.449698 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.449683    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.507027 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.506991    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.507178 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.507084    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wclfx\" (UniqueName: \"kubernetes.io/projected/66c1fd2b-2b0b-40fc-8e19-fcb2602498a4-kube-api-access-wclfx\") pod \"maas-default-gateway-openshift-default-855f9574cf-plj6l\" (UID: \"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.510933 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.510916    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:24.708982 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.708951    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:00:24.939232 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.939159    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"]
Nov 01 07:00:24.941503 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:00:24.941478    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c1fd2b_2b0b_40fc_8e19_fcb2602498a4.slice/crio-d79a9b5af06b4e5c8d2f0d9285d833ca3a8264d90c4c07a25eb08388f23cc9d9 WatchSource:0}: Error finding container d79a9b5af06b4e5c8d2f0d9285d833ca3a8264d90c4c07a25eb08388f23cc9d9: Status 404 returned error can't find the container with id d79a9b5af06b4e5c8d2f0d9285d833ca3a8264d90c4c07a25eb08388f23cc9d9
Nov 01 07:00:24.943584 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.943556    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:00:24.943636 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.943619    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:00:24.943671 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:24.943664    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:00:25.322276 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:25.322199    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" event={"ID":"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4","Type":"ContainerStarted","Data":"c9464d48be22d50f5375e23afc16d15a85fa0d5202a576b34e1b53d40fab1aba"}
Nov 01 07:00:25.322276 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:25.322254    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" event={"ID":"66c1fd2b-2b0b-40fc-8e19-fcb2602498a4","Type":"ContainerStarted","Data":"d79a9b5af06b4e5c8d2f0d9285d833ca3a8264d90c4c07a25eb08388f23cc9d9"}
Nov 01 07:00:25.445558 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:25.445500    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podStartSLOduration=1.445486041 podStartE2EDuration="1.445486041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:25.443425439 +0000 UTC m=+4462.321398863" watchObservedRunningTime="2026-11-01 07:00:25.445486041 +0000 UTC m=+4462.323459452"
Nov 01 07:00:25.512024 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:25.511991    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:25.513565 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:25.513539    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 01 07:00:25.513683 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:25.513627    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 01 07:00:26.512176 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:26.512137    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 01 07:00:26.512688 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:26.512199    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 01 07:00:27.511943 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:27.511908    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 01 07:00:27.512128 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:27.511958    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 01 07:00:28.511884 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:28.511818    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 01 07:00:28.512395 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:28.511911    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 01 07:00:29.511611 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:29.511580    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 01 07:00:29.511822 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:29.511632    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 01 07:00:30.511598 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:30.511563    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 01 07:00:30.512125 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:30.511640    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 01 07:00:31.511531 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:31.511494    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 01 07:00:31.511726 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:31.511561    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 01 07:00:32.511730 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:32.511692    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 01 07:00:32.512158 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:32.511773    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 01 07:00:33.516071 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:33.516043    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:33.516551 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:33.516437    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:33.517184 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:33.517159    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l"
Nov 01 07:00:47.545339 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:00:47.545295    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:00:47.615508 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:47.615478    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:00:54.986768 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:54.986732    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"]
Nov 01 07:00:54.990380 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:54.990363    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.042765 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.042742    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:00:55.071303 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.071273    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.071445 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.071306    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.071445 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.071343    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb5r7\" (UniqueName: \"kubernetes.io/projected/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-kube-api-access-lb5r7\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.092353 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.092327    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:00:55.092353 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.092347    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:00:55.092527 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.092512    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:00:55.094040 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.094023    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"]
Nov 01 07:00:55.172481 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.172455    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.172639 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.172483    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.172639 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.172514    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5r7\" (UniqueName: \"kubernetes.io/projected/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-kube-api-access-lb5r7\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.173178 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.173155    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.175001 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.174975    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.233736 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.233711    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb5r7\" (UniqueName: \"kubernetes.io/projected/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-kube-api-access-lb5r7\") pod \"maas-api-revoke-sub-5bfc65cc-jf8vn\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.303733 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.303654    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:55.678592 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:55.678518    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"]
Nov 01 07:00:55.680947 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:00:55.680921    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a324ef6_d72c_4c7b_9f8a_f5339e7f0e06.slice/crio-fb8e303031aa3a96a9a53046075959204bc845c5d88af880791ebc5765ad5cbe WatchSource:0}: Error finding container fb8e303031aa3a96a9a53046075959204bc845c5d88af880791ebc5765ad5cbe: Status 404 returned error can't find the container with id fb8e303031aa3a96a9a53046075959204bc845c5d88af880791ebc5765ad5cbe
Nov 01 07:00:56.526033 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:56.525998    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn" event={"ID":"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06","Type":"ContainerStarted","Data":"241120fe1e402fbdcd4e6549d8e958f31d7dccec45719d300ea293e79a8b8a07"}
Nov 01 07:00:56.526397 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:56.526055    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn" event={"ID":"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06","Type":"ContainerStarted","Data":"fb8e303031aa3a96a9a53046075959204bc845c5d88af880791ebc5765ad5cbe"}
Nov 01 07:00:56.642037 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:56.641990    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn" podStartSLOduration=2.641975876 podStartE2EDuration="2.641975876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:56.64076005 +0000 UTC m=+4493.518733461" watchObservedRunningTime="2026-11-01 07:00:56.641975876 +0000 UTC m=+4493.519949283"
Nov 01 07:00:57.531860 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:57.531810    5510 generic.go:356] "Generic (PLEG): container finished" podID="6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06" containerID="241120fe1e402fbdcd4e6549d8e958f31d7dccec45719d300ea293e79a8b8a07" exitCode=0
Nov 01 07:00:57.532307 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:57.531899    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn" event={"ID":"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06","Type":"ContainerDied","Data":"241120fe1e402fbdcd4e6549d8e958f31d7dccec45719d300ea293e79a8b8a07"}
Nov 01 07:00:58.672799 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.672768    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:58.808080 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.808016    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-openshift-ca\") pod \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") "
Nov 01 07:00:58.808080 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.808050    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-kube-api-access-lb5r7\" (UniqueName: \"kubernetes.io/projected/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-kube-api-access-lb5r7\") pod \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") "
Nov 01 07:00:58.808256 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.808213    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-xks-ca\") pod \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\" (UID: \"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06\") "
Nov 01 07:00:58.808426 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.808405    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-openshift-ca" pod "6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06" (UID: "6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:58.808543 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.808528    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:58.810445 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.810418    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-xks-ca" pod "6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06" (UID: "6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:00:58.810575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.810562    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-kube-api-access-lb5r7" pod "6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06" (UID: "6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06"). InnerVolumeSpecName "kube-api-access-lb5r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:58.909677 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.909643    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:58.909677 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:58.909672    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb5r7\" (UniqueName: \"kubernetes.io/projected/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06-kube-api-access-lb5r7\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:59.542653 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:59.542569    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"
Nov 01 07:00:59.542653 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:59.542597    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn" event={"ID":"6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06","Type":"ContainerDied","Data":"fb8e303031aa3a96a9a53046075959204bc845c5d88af880791ebc5765ad5cbe"}
Nov 01 07:00:59.542653 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:00:59.542647    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb8e303031aa3a96a9a53046075959204bc845c5d88af880791ebc5765ad5cbe"
Nov 01 07:01:06.009887 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.009853    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"]
Nov 01 07:01:06.010266 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.010228    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06" containerName="revoke-keys"
Nov 01 07:01:06.010266 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.010238    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06" containerName="revoke-keys"
Nov 01 07:01:06.010339 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.010335    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06" containerName="revoke-keys"
Nov 01 07:01:06.014540 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.014523    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.081634 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.081601    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2kh\" (UniqueName: \"kubernetes.io/projected/cd3828f0-cef8-4c63-97fb-ba04539606b0-kube-api-access-vl2kh\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.081811 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.081724    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.081811 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.081774    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.118467 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.118433    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:01:06.118467 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.118448    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:01:06.118641 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.118478    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:01:06.118641 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.118507    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:01:06.120250 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.120229    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"]
Nov 01 07:01:06.182173 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.182147    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.182304 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.182208    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2kh\" (UniqueName: \"kubernetes.io/projected/cd3828f0-cef8-4c63-97fb-ba04539606b0-kube-api-access-vl2kh\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.182304 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.182279    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.183026 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.182956    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.184640 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.184622    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.243606 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.243582    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2kh\" (UniqueName: \"kubernetes.io/projected/cd3828f0-cef8-4c63-97fb-ba04539606b0-kube-api-access-vl2kh\") pod \"maas-api-revoke-sub-69f7e4b7-cgdrl\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.326653 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.326589    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:06.698412 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:06.698388    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"]
Nov 01 07:01:06.700335 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:01:06.700306    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3828f0_cef8_4c63_97fb_ba04539606b0.slice/crio-837be643998c3ade67f495cace1318f354f0e2bc4fb2336e65da90cdc9893401 WatchSource:0}: Error finding container 837be643998c3ade67f495cace1318f354f0e2bc4fb2336e65da90cdc9893401: Status 404 returned error can't find the container with id 837be643998c3ade67f495cace1318f354f0e2bc4fb2336e65da90cdc9893401
Nov 01 07:01:07.583208 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:07.583174    5510 generic.go:356] "Generic (PLEG): container finished" podID="cd3828f0-cef8-4c63-97fb-ba04539606b0" containerID="b0d686d861e7861a3db81fbb0fd3793c1ca91088abd33ba5d721b88be6a924d5" exitCode=0
Nov 01 07:01:07.583626 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:07.583257    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl" event={"ID":"cd3828f0-cef8-4c63-97fb-ba04539606b0","Type":"ContainerDied","Data":"b0d686d861e7861a3db81fbb0fd3793c1ca91088abd33ba5d721b88be6a924d5"}
Nov 01 07:01:07.583626 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:07.583312    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl" event={"ID":"cd3828f0-cef8-4c63-97fb-ba04539606b0","Type":"ContainerStarted","Data":"837be643998c3ade67f495cace1318f354f0e2bc4fb2336e65da90cdc9893401"}
Nov 01 07:01:08.721656 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.721634    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:08.806320 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.806295    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3828f0-cef8-4c63-97fb-ba04539606b0-kube-api-access-vl2kh\" (UniqueName: \"kubernetes.io/projected/cd3828f0-cef8-4c63-97fb-ba04539606b0-kube-api-access-vl2kh\") pod \"cd3828f0-cef8-4c63-97fb-ba04539606b0\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") "
Nov 01 07:01:08.806466 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.806357    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-openshift-ca\") pod \"cd3828f0-cef8-4c63-97fb-ba04539606b0\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") "
Nov 01 07:01:08.806466 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.806399    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-xks-ca\") pod \"cd3828f0-cef8-4c63-97fb-ba04539606b0\" (UID: \"cd3828f0-cef8-4c63-97fb-ba04539606b0\") "
Nov 01 07:01:08.806805 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.806765    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-openshift-ca" pod "cd3828f0-cef8-4c63-97fb-ba04539606b0" (UID: "cd3828f0-cef8-4c63-97fb-ba04539606b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:01:08.808514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.808492    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3828f0-cef8-4c63-97fb-ba04539606b0-kube-api-access-vl2kh" pod "cd3828f0-cef8-4c63-97fb-ba04539606b0" (UID: "cd3828f0-cef8-4c63-97fb-ba04539606b0"). InnerVolumeSpecName "kube-api-access-vl2kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:08.808593 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.808547    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-xks-ca" pod "cd3828f0-cef8-4c63-97fb-ba04539606b0" (UID: "cd3828f0-cef8-4c63-97fb-ba04539606b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:01:08.908223 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.908136    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:08.908223 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.908168    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3828f0-cef8-4c63-97fb-ba04539606b0-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:08.908223 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:08.908183    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2kh\" (UniqueName: \"kubernetes.io/projected/cd3828f0-cef8-4c63-97fb-ba04539606b0-kube-api-access-vl2kh\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:09.594755 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:09.594680    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"
Nov 01 07:01:09.594755 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:09.594725    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl" event={"ID":"cd3828f0-cef8-4c63-97fb-ba04539606b0","Type":"ContainerDied","Data":"837be643998c3ade67f495cace1318f354f0e2bc4fb2336e65da90cdc9893401"}
Nov 01 07:01:09.595012 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:09.594768    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="837be643998c3ade67f495cace1318f354f0e2bc4fb2336e65da90cdc9893401"
Nov 01 07:01:10.094404 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:10.094369    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:01:16.899506 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:16.899435    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"]
Nov 01 07:01:16.899923 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:16.899902    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cd3828f0-cef8-4c63-97fb-ba04539606b0" containerName="revoke-keys"
Nov 01 07:01:16.899923 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:16.899917    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3828f0-cef8-4c63-97fb-ba04539606b0" containerName="revoke-keys"
Nov 01 07:01:16.900015 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:16.900003    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3828f0-cef8-4c63-97fb-ba04539606b0" containerName="revoke-keys"
Nov 01 07:01:16.903161 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:16.903146    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.005502 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.005462    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:01:17.005502 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.005462    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:01:17.005739 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.005642    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:01:17.005739 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.005675    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:01:17.007281 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.007264    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"]
Nov 01 07:01:17.010085 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.010067    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.010141 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.010110    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7gs7\" (UniqueName: \"kubernetes.io/projected/fb67b6a3-bc08-4eab-bfda-d50821eeb947-kube-api-access-j7gs7\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.010207 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.010189    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.111702 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.111669    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.111876 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.111725    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7gs7\" (UniqueName: \"kubernetes.io/projected/fb67b6a3-bc08-4eab-bfda-d50821eeb947-kube-api-access-j7gs7\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.111876 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.111768    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.112496 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.112477    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.114257 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.114237    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.172372 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.172309    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7gs7\" (UniqueName: \"kubernetes.io/projected/fb67b6a3-bc08-4eab-bfda-d50821eeb947-kube-api-access-j7gs7\") pod \"maas-api-revoke-sub-fec2b1d6-5jcks\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.214920 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.214892    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:17.544270 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:01:17.544229    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:01:17.586572 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.586550    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"]
Nov 01 07:01:17.588293 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:01:17.588269    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb67b6a3_bc08_4eab_bfda_d50821eeb947.slice/crio-c9a3a33da357ab90a33f47f49c231c93b90ddb5ec3dceb4a9796a2d1818ca3cd WatchSource:0}: Error finding container c9a3a33da357ab90a33f47f49c231c93b90ddb5ec3dceb4a9796a2d1818ca3cd: Status 404 returned error can't find the container with id c9a3a33da357ab90a33f47f49c231c93b90ddb5ec3dceb4a9796a2d1818ca3cd
Nov 01 07:01:17.654151 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:17.654124    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks" event={"ID":"fb67b6a3-bc08-4eab-bfda-d50821eeb947","Type":"ContainerStarted","Data":"c9a3a33da357ab90a33f47f49c231c93b90ddb5ec3dceb4a9796a2d1818ca3cd"}
Nov 01 07:01:18.660050 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:18.660011    5510 generic.go:356] "Generic (PLEG): container finished" podID="fb67b6a3-bc08-4eab-bfda-d50821eeb947" containerID="6db88592b2d840da9cfd0fd301297b79035526050de029072c080c7d7587fe93" exitCode=0
Nov 01 07:01:18.660487 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:18.660054    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks" event={"ID":"fb67b6a3-bc08-4eab-bfda-d50821eeb947","Type":"ContainerDied","Data":"6db88592b2d840da9cfd0fd301297b79035526050de029072c080c7d7587fe93"}
Nov 01 07:01:19.798898 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.798875    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:19.841241 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.841215    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-xks-ca\") pod \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") "
Nov 01 07:01:19.841394 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.841264    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb67b6a3-bc08-4eab-bfda-d50821eeb947-kube-api-access-j7gs7\" (UniqueName: \"kubernetes.io/projected/fb67b6a3-bc08-4eab-bfda-d50821eeb947-kube-api-access-j7gs7\") pod \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") "
Nov 01 07:01:19.841394 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.841339    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-openshift-ca\") pod \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\" (UID: \"fb67b6a3-bc08-4eab-bfda-d50821eeb947\") "
Nov 01 07:01:19.841703 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.841681    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-openshift-ca" pod "fb67b6a3-bc08-4eab-bfda-d50821eeb947" (UID: "fb67b6a3-bc08-4eab-bfda-d50821eeb947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:01:19.843515 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.843493    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-xks-ca" pod "fb67b6a3-bc08-4eab-bfda-d50821eeb947" (UID: "fb67b6a3-bc08-4eab-bfda-d50821eeb947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:01:19.843593 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.843500    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb67b6a3-bc08-4eab-bfda-d50821eeb947-kube-api-access-j7gs7" pod "fb67b6a3-bc08-4eab-bfda-d50821eeb947" (UID: "fb67b6a3-bc08-4eab-bfda-d50821eeb947"). InnerVolumeSpecName "kube-api-access-j7gs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:19.942185 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.942123    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:19.942185 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.942149    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb67b6a3-bc08-4eab-bfda-d50821eeb947-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:19.942185 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:19.942158    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7gs7\" (UniqueName: \"kubernetes.io/projected/fb67b6a3-bc08-4eab-bfda-d50821eeb947-kube-api-access-j7gs7\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:20.670474 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:20.670436    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks" event={"ID":"fb67b6a3-bc08-4eab-bfda-d50821eeb947","Type":"ContainerDied","Data":"c9a3a33da357ab90a33f47f49c231c93b90ddb5ec3dceb4a9796a2d1818ca3cd"}
Nov 01 07:01:20.670474 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:20.670470    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"
Nov 01 07:01:20.670682 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:20.670493    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a3a33da357ab90a33f47f49c231c93b90ddb5ec3dceb4a9796a2d1818ca3cd"
Nov 01 07:01:27.597796 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.597759    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"]
Nov 01 07:01:27.598317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.598213    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fb67b6a3-bc08-4eab-bfda-d50821eeb947" containerName="revoke-keys"
Nov 01 07:01:27.598317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.598225    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb67b6a3-bc08-4eab-bfda-d50821eeb947" containerName="revoke-keys"
Nov 01 07:01:27.598317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.598312    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb67b6a3-bc08-4eab-bfda-d50821eeb947" containerName="revoke-keys"
Nov 01 07:01:27.601605 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.601582    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.653395 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.653377    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:01:27.673910 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.673882    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.674054 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.673915    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdm9t\" (UniqueName: \"kubernetes.io/projected/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-kube-api-access-cdm9t\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.674054 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.673940    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.702971 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.702944    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:01:27.702971 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.702944    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:01:27.703128 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.702955    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:01:27.704875 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.704854    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"]
Nov 01 07:01:27.774955 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.774925    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.775107 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.774961    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdm9t\" (UniqueName: \"kubernetes.io/projected/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-kube-api-access-cdm9t\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.775107 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.774993    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.775542 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.775517    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.777424 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.777406    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.835493 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.835461    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdm9t\" (UniqueName: \"kubernetes.io/projected/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-kube-api-access-cdm9t\") pod \"maas-api-revoke-sub-d3b2d386-cfqmm\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:27.914030 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:27.913957    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:28.286168 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:28.286141    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"]
Nov 01 07:01:28.287954 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:01:28.287926    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7eb1ea3_87e3_4c7c_a786_d4efa8838659.slice/crio-21fa697134da4450bf04f7e117abd3dc2e0b239048683080e0176c4f7d22773e WatchSource:0}: Error finding container 21fa697134da4450bf04f7e117abd3dc2e0b239048683080e0176c4f7d22773e: Status 404 returned error can't find the container with id 21fa697134da4450bf04f7e117abd3dc2e0b239048683080e0176c4f7d22773e
Nov 01 07:01:28.795898 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:28.795859    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm" event={"ID":"e7eb1ea3-87e3-4c7c-a786-d4efa8838659","Type":"ContainerStarted","Data":"5a90a642611f6bc61113df2e33693745089b6b09d6bea85d81c2aae97a50392e"}
Nov 01 07:01:28.796277 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:28.795920    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm" event={"ID":"e7eb1ea3-87e3-4c7c-a786-d4efa8838659","Type":"ContainerStarted","Data":"21fa697134da4450bf04f7e117abd3dc2e0b239048683080e0176c4f7d22773e"}
Nov 01 07:01:28.911579 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:28.911523    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm" podStartSLOduration=1.91151096 podStartE2EDuration="1.91151096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:01:28.910013802 +0000 UTC m=+4525.787987222" watchObservedRunningTime="2026-11-01 07:01:28.91151096 +0000 UTC m=+4525.789484369"
Nov 01 07:01:29.802478 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:29.802442    5510 generic.go:356] "Generic (PLEG): container finished" podID="e7eb1ea3-87e3-4c7c-a786-d4efa8838659" containerID="5a90a642611f6bc61113df2e33693745089b6b09d6bea85d81c2aae97a50392e" exitCode=0
Nov 01 07:01:29.802478 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:29.802478    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm" event={"ID":"e7eb1ea3-87e3-4c7c-a786-d4efa8838659","Type":"ContainerDied","Data":"5a90a642611f6bc61113df2e33693745089b6b09d6bea85d81c2aae97a50392e"}
Nov 01 07:01:30.943302 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:30.943276    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:31.003472 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.003442    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-kube-api-access-cdm9t\" (UniqueName: \"kubernetes.io/projected/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-kube-api-access-cdm9t\") pod \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") "
Nov 01 07:01:31.003472 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.003477    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-openshift-ca\") pod \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") "
Nov 01 07:01:31.003662 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.003510    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-xks-ca\") pod \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\" (UID: \"e7eb1ea3-87e3-4c7c-a786-d4efa8838659\") "
Nov 01 07:01:31.003915 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.003892    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-openshift-ca" pod "e7eb1ea3-87e3-4c7c-a786-d4efa8838659" (UID: "e7eb1ea3-87e3-4c7c-a786-d4efa8838659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:01:31.005657 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.005633    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-kube-api-access-cdm9t" pod "e7eb1ea3-87e3-4c7c-a786-d4efa8838659" (UID: "e7eb1ea3-87e3-4c7c-a786-d4efa8838659"). InnerVolumeSpecName "kube-api-access-cdm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:31.005791 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.005773    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-xks-ca" pod "e7eb1ea3-87e3-4c7c-a786-d4efa8838659" (UID: "e7eb1ea3-87e3-4c7c-a786-d4efa8838659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:01:31.104286 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.104211    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdm9t\" (UniqueName: \"kubernetes.io/projected/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-kube-api-access-cdm9t\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:31.104286 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.104238    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:31.104286 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.104249    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7eb1ea3-87e3-4c7c-a786-d4efa8838659-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:31.813283 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.813256    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"
Nov 01 07:01:31.813471 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.813248    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm" event={"ID":"e7eb1ea3-87e3-4c7c-a786-d4efa8838659","Type":"ContainerDied","Data":"21fa697134da4450bf04f7e117abd3dc2e0b239048683080e0176c4f7d22773e"}
Nov 01 07:01:31.813471 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:31.813406    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21fa697134da4450bf04f7e117abd3dc2e0b239048683080e0176c4f7d22773e"
Nov 01 07:01:38.649116 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.649087    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"]
Nov 01 07:01:38.649514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.649476    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e7eb1ea3-87e3-4c7c-a786-d4efa8838659" containerName="revoke-keys"
Nov 01 07:01:38.649514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.649486    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7eb1ea3-87e3-4c7c-a786-d4efa8838659" containerName="revoke-keys"
Nov 01 07:01:38.649596 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.649569    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7eb1ea3-87e3-4c7c-a786-d4efa8838659" containerName="revoke-keys"
Nov 01 07:01:38.652631 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.652617    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.704533 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.704506    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:01:38.753882 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.753857    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:01:38.754042 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.753907    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:01:38.754042 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.753972    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:01:38.756066 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.756049    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"]
Nov 01 07:01:38.775879 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.775853    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.776006 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.775898    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g74t\" (UniqueName: \"kubernetes.io/projected/2bf5b3ab-7f65-4106-93e8-fc556868feb9-kube-api-access-9g74t\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.776006 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.775988    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.877325 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.877291    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.877525 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.877349    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g74t\" (UniqueName: \"kubernetes.io/projected/2bf5b3ab-7f65-4106-93e8-fc556868feb9-kube-api-access-9g74t\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.877525 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.877395    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.878101 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.878076    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.879734 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.879715    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.938121 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.938053    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g74t\" (UniqueName: \"kubernetes.io/projected/2bf5b3ab-7f65-4106-93e8-fc556868feb9-kube-api-access-9g74t\") pod \"maas-api-revoke-sub-eff1785a-qrzvp\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:38.966008 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:38.965981    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:39.339496 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:39.339465    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"]
Nov 01 07:01:39.342276 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:01:39.342248    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf5b3ab_7f65_4106_93e8_fc556868feb9.slice/crio-c72f96521fd924eec5af65cd0d85aae2c4b7be95999679d94e82ad275e7e713a WatchSource:0}: Error finding container c72f96521fd924eec5af65cd0d85aae2c4b7be95999679d94e82ad275e7e713a: Status 404 returned error can't find the container with id c72f96521fd924eec5af65cd0d85aae2c4b7be95999679d94e82ad275e7e713a
Nov 01 07:01:39.850728 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:39.850639    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp" event={"ID":"2bf5b3ab-7f65-4106-93e8-fc556868feb9","Type":"ContainerStarted","Data":"f86795913b1831d428fe717fd9ebbe12cea995b25a6de29e95c2250152d61b70"}
Nov 01 07:01:39.850728 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:39.850694    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp" event={"ID":"2bf5b3ab-7f65-4106-93e8-fc556868feb9","Type":"ContainerStarted","Data":"c72f96521fd924eec5af65cd0d85aae2c4b7be95999679d94e82ad275e7e713a"}
Nov 01 07:01:39.966822 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:39.966768    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp" podStartSLOduration=1.966754 podStartE2EDuration="1.966754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:01:39.964856522 +0000 UTC m=+4536.842829932" watchObservedRunningTime="2026-11-01 07:01:39.966754 +0000 UTC m=+4536.844727404"
Nov 01 07:01:40.857266 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:40.857234    5510 generic.go:356] "Generic (PLEG): container finished" podID="2bf5b3ab-7f65-4106-93e8-fc556868feb9" containerID="f86795913b1831d428fe717fd9ebbe12cea995b25a6de29e95c2250152d61b70" exitCode=0
Nov 01 07:01:40.857739 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:40.857273    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp" event={"ID":"2bf5b3ab-7f65-4106-93e8-fc556868feb9","Type":"ContainerDied","Data":"f86795913b1831d428fe717fd9ebbe12cea995b25a6de29e95c2250152d61b70"}
Nov 01 07:01:42.059714 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.059681    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:42.110961 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.110926    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-openshift-ca\") pod \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") "
Nov 01 07:01:42.111130 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.110998    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-xks-ca\") pod \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") "
Nov 01 07:01:42.111205 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.111190    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf5b3ab-7f65-4106-93e8-fc556868feb9-kube-api-access-9g74t\" (UniqueName: \"kubernetes.io/projected/2bf5b3ab-7f65-4106-93e8-fc556868feb9-kube-api-access-9g74t\") pod \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\" (UID: \"2bf5b3ab-7f65-4106-93e8-fc556868feb9\") "
Nov 01 07:01:42.111739 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.111704    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-openshift-ca" pod "2bf5b3ab-7f65-4106-93e8-fc556868feb9" (UID: "2bf5b3ab-7f65-4106-93e8-fc556868feb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:01:42.113866 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.113812    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-xks-ca" pod "2bf5b3ab-7f65-4106-93e8-fc556868feb9" (UID: "2bf5b3ab-7f65-4106-93e8-fc556868feb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:01:42.114198 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.114171    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf5b3ab-7f65-4106-93e8-fc556868feb9-kube-api-access-9g74t" pod "2bf5b3ab-7f65-4106-93e8-fc556868feb9" (UID: "2bf5b3ab-7f65-4106-93e8-fc556868feb9"). InnerVolumeSpecName "kube-api-access-9g74t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:42.253213 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.252403    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:42.253213 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.252447    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf5b3ab-7f65-4106-93e8-fc556868feb9-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:42.253213 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.252466    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g74t\" (UniqueName: \"kubernetes.io/projected/2bf5b3ab-7f65-4106-93e8-fc556868feb9-kube-api-access-9g74t\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:42.936850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.933502    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp" event={"ID":"2bf5b3ab-7f65-4106-93e8-fc556868feb9","Type":"ContainerDied","Data":"c72f96521fd924eec5af65cd0d85aae2c4b7be95999679d94e82ad275e7e713a"}
Nov 01 07:01:42.936850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.933567    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c72f96521fd924eec5af65cd0d85aae2c4b7be95999679d94e82ad275e7e713a"
Nov 01 07:01:42.936850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:42.933652    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"
Nov 01 07:01:47.544402 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:01:47.544356    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:01:49.655934 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.655903    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"]
Nov 01 07:01:49.656363 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.656298    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2bf5b3ab-7f65-4106-93e8-fc556868feb9" containerName="revoke-keys"
Nov 01 07:01:49.656363 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.656309    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf5b3ab-7f65-4106-93e8-fc556868feb9" containerName="revoke-keys"
Nov 01 07:01:49.656454 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.656401    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf5b3ab-7f65-4106-93e8-fc556868feb9" containerName="revoke-keys"
Nov 01 07:01:49.659266 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.659250    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.711151 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.711121    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:01:49.760615 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.760588    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:01:49.760910 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.760892    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:01:49.761003 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.760966    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:01:49.762678 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.762661    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"]
Nov 01 07:01:49.805003 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.804975    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.805158 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.805018    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.805158 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.805105    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmfvx\" (UniqueName: \"kubernetes.io/projected/b4261dfb-4974-436f-9726-c5516fc9a02c-kube-api-access-fmfvx\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.906626 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.906547    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.906626 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.906608    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.906857 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.906650    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmfvx\" (UniqueName: \"kubernetes.io/projected/b4261dfb-4974-436f-9726-c5516fc9a02c-kube-api-access-fmfvx\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.907398 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.907314    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.909083 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.909060    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.967084 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.967060    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmfvx\" (UniqueName: \"kubernetes.io/projected/b4261dfb-4974-436f-9726-c5516fc9a02c-kube-api-access-fmfvx\") pod \"maas-api-revoke-sub-7bc2b22e-wfrjm\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:49.972882 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:49.972866    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:50.343224 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:50.343191    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"]
Nov 01 07:01:50.345665 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:01:50.345633    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4261dfb_4974_436f_9726_c5516fc9a02c.slice/crio-04902a9b6d96f7a103b621fedeacd3d25baa411e02abdfb0007c840449d649e3 WatchSource:0}: Error finding container 04902a9b6d96f7a103b621fedeacd3d25baa411e02abdfb0007c840449d649e3: Status 404 returned error can't find the container with id 04902a9b6d96f7a103b621fedeacd3d25baa411e02abdfb0007c840449d649e3
Nov 01 07:01:50.986645 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:50.986601    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm" event={"ID":"b4261dfb-4974-436f-9726-c5516fc9a02c","Type":"ContainerStarted","Data":"5ff3fffaf927fd72240a113db7931203abc199d9de51b8911c571f8e66f8f088"}
Nov 01 07:01:50.987040 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:50.986668    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm" event={"ID":"b4261dfb-4974-436f-9726-c5516fc9a02c","Type":"ContainerStarted","Data":"04902a9b6d96f7a103b621fedeacd3d25baa411e02abdfb0007c840449d649e3"}
Nov 01 07:01:51.106341 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:51.103154    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm" podStartSLOduration=2.10313387 podStartE2EDuration="2.10313387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:01:51.101034136 +0000 UTC m=+4547.979007546" watchObservedRunningTime="2026-11-01 07:01:51.10313387 +0000 UTC m=+4547.981107281"
Nov 01 07:01:51.992701 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:51.992666    5510 generic.go:356] "Generic (PLEG): container finished" podID="b4261dfb-4974-436f-9726-c5516fc9a02c" containerID="5ff3fffaf927fd72240a113db7931203abc199d9de51b8911c571f8e66f8f088" exitCode=0
Nov 01 07:01:51.993131 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:51.992753    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm" event={"ID":"b4261dfb-4974-436f-9726-c5516fc9a02c","Type":"ContainerDied","Data":"5ff3fffaf927fd72240a113db7931203abc199d9de51b8911c571f8e66f8f088"}
Nov 01 07:01:53.215728 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.215685    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:01:53.342686 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.342594    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-xks-ca\") pod \"b4261dfb-4974-436f-9726-c5516fc9a02c\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") "
Nov 01 07:01:53.342686 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.342647    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4261dfb-4974-436f-9726-c5516fc9a02c-kube-api-access-fmfvx\" (UniqueName: \"kubernetes.io/projected/b4261dfb-4974-436f-9726-c5516fc9a02c-kube-api-access-fmfvx\") pod \"b4261dfb-4974-436f-9726-c5516fc9a02c\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") "
Nov 01 07:01:53.342686 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.342678    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-openshift-ca\") pod \"b4261dfb-4974-436f-9726-c5516fc9a02c\" (UID: \"b4261dfb-4974-436f-9726-c5516fc9a02c\") "
Nov 01 07:01:53.343273 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.343249    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-openshift-ca" pod "b4261dfb-4974-436f-9726-c5516fc9a02c" (UID: "b4261dfb-4974-436f-9726-c5516fc9a02c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:01:53.346003 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.345979    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4261dfb-4974-436f-9726-c5516fc9a02c-kube-api-access-fmfvx" pod "b4261dfb-4974-436f-9726-c5516fc9a02c" (UID: "b4261dfb-4974-436f-9726-c5516fc9a02c"). InnerVolumeSpecName "kube-api-access-fmfvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:53.346183 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.346164    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-xks-ca" pod "b4261dfb-4974-436f-9726-c5516fc9a02c" (UID: "b4261dfb-4974-436f-9726-c5516fc9a02c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:01:53.450554 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.450462    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:53.450554 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.450507    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmfvx\" (UniqueName: \"kubernetes.io/projected/b4261dfb-4974-436f-9726-c5516fc9a02c-kube-api-access-fmfvx\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:53.450554 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:53.450525    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4261dfb-4974-436f-9726-c5516fc9a02c-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:54.022345 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:54.022305    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm" event={"ID":"b4261dfb-4974-436f-9726-c5516fc9a02c","Type":"ContainerDied","Data":"04902a9b6d96f7a103b621fedeacd3d25baa411e02abdfb0007c840449d649e3"}
Nov 01 07:01:54.022677 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:54.022659    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04902a9b6d96f7a103b621fedeacd3d25baa411e02abdfb0007c840449d649e3"
Nov 01 07:01:54.032850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:01:54.022608    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"
Nov 01 07:02:00.456002 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.455970    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"]
Nov 01 07:02:00.456498 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.456372    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b4261dfb-4974-436f-9726-c5516fc9a02c" containerName="revoke-keys"
Nov 01 07:02:00.456498 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.456382    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4261dfb-4974-436f-9726-c5516fc9a02c" containerName="revoke-keys"
Nov 01 07:02:00.456498 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.456470    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4261dfb-4974-436f-9726-c5516fc9a02c" containerName="revoke-keys"
Nov 01 07:02:00.459813 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.459796    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.511656 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.511630    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:02:00.561219 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.561188    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:02:00.561374 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.561223    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:02:00.561374 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.561221    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:02:00.563133 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.563114    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"]
Nov 01 07:02:00.572765 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.572741    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxg9\" (UniqueName: \"kubernetes.io/projected/36f605c2-6b4d-454f-9a08-93c15644877c-kube-api-access-6bxg9\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.572879 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.572775    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.572995 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.572889    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.674045 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.674014    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.674183 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.674114    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxg9\" (UniqueName: \"kubernetes.io/projected/36f605c2-6b4d-454f-9a08-93c15644877c-kube-api-access-6bxg9\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.674183 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.674142    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.674651 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.674634    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.676512 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.676488    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.736365 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.736337    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxg9\" (UniqueName: \"kubernetes.io/projected/36f605c2-6b4d-454f-9a08-93c15644877c-kube-api-access-6bxg9\") pod \"maas-api-revoke-sub-f25cc81f-z5p7h\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:00.772821 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:00.772784    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:01.144639 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:01.144614    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"]
Nov 01 07:02:01.146877 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:02:01.146853    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f605c2_6b4d_454f_9a08_93c15644877c.slice/crio-ad0597170e140cabd9bfb797ce410afaa7581e705ca81dec135235099f116d79 WatchSource:0}: Error finding container ad0597170e140cabd9bfb797ce410afaa7581e705ca81dec135235099f116d79: Status 404 returned error can't find the container with id ad0597170e140cabd9bfb797ce410afaa7581e705ca81dec135235099f116d79
Nov 01 07:02:02.075510 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:02.075482    5510 generic.go:356] "Generic (PLEG): container finished" podID="36f605c2-6b4d-454f-9a08-93c15644877c" containerID="e53be8a835ba0694a8caf589e4447bff06e235f6c233f2e28cf9faaf8c727745" exitCode=0
Nov 01 07:02:02.075903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:02.075568    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h" event={"ID":"36f605c2-6b4d-454f-9a08-93c15644877c","Type":"ContainerDied","Data":"e53be8a835ba0694a8caf589e4447bff06e235f6c233f2e28cf9faaf8c727745"}
Nov 01 07:02:02.075903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:02.075621    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h" event={"ID":"36f605c2-6b4d-454f-9a08-93c15644877c","Type":"ContainerStarted","Data":"ad0597170e140cabd9bfb797ce410afaa7581e705ca81dec135235099f116d79"}
Nov 01 07:02:03.220173 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.220149    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:03.302590 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.302563    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-openshift-ca\") pod \"36f605c2-6b4d-454f-9a08-93c15644877c\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") "
Nov 01 07:02:03.302752 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.302633    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f605c2-6b4d-454f-9a08-93c15644877c-kube-api-access-6bxg9\" (UniqueName: \"kubernetes.io/projected/36f605c2-6b4d-454f-9a08-93c15644877c-kube-api-access-6bxg9\") pod \"36f605c2-6b4d-454f-9a08-93c15644877c\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") "
Nov 01 07:02:03.302752 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.302667    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-xks-ca\") pod \"36f605c2-6b4d-454f-9a08-93c15644877c\" (UID: \"36f605c2-6b4d-454f-9a08-93c15644877c\") "
Nov 01 07:02:03.302967 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.302942    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-openshift-ca" pod "36f605c2-6b4d-454f-9a08-93c15644877c" (UID: "36f605c2-6b4d-454f-9a08-93c15644877c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:02:03.304898 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.304875    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-xks-ca" pod "36f605c2-6b4d-454f-9a08-93c15644877c" (UID: "36f605c2-6b4d-454f-9a08-93c15644877c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:02:03.304962 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.304942    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f605c2-6b4d-454f-9a08-93c15644877c-kube-api-access-6bxg9" pod "36f605c2-6b4d-454f-9a08-93c15644877c" (UID: "36f605c2-6b4d-454f-9a08-93c15644877c"). InnerVolumeSpecName "kube-api-access-6bxg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:03.405556 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.404570    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:03.405556 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.404612    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxg9\" (UniqueName: \"kubernetes.io/projected/36f605c2-6b4d-454f-9a08-93c15644877c-kube-api-access-6bxg9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:03.405556 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:03.404628    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f605c2-6b4d-454f-9a08-93c15644877c-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:04.090234 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:04.090195    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h" event={"ID":"36f605c2-6b4d-454f-9a08-93c15644877c","Type":"ContainerDied","Data":"ad0597170e140cabd9bfb797ce410afaa7581e705ca81dec135235099f116d79"}
Nov 01 07:02:04.090234 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:04.090226    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"
Nov 01 07:02:04.090523 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:04.090261    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad0597170e140cabd9bfb797ce410afaa7581e705ca81dec135235099f116d79"
Nov 01 07:02:10.048761 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:10.048728    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"]
Nov 01 07:02:10.098417 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:10.098383    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-95cqs"]
Nov 01 07:02:11.234652 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.234617    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a971632d-0719-4b78-b684-6ce7e5931532" path="/var/lib/kubelet/pods/a971632d-0719-4b78-b684-6ce7e5931532/volumes"
Nov 01 07:02:11.357850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.357801    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"]
Nov 01 07:02:11.358353 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.358333    5510 cpu_manager.go:414] "containerMap: removing container" podUID="36f605c2-6b4d-454f-9a08-93c15644877c" containerName="revoke-keys"
Nov 01 07:02:11.358487 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.358355    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f605c2-6b4d-454f-9a08-93c15644877c" containerName="revoke-keys"
Nov 01 07:02:11.358487 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.358459    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f605c2-6b4d-454f-9a08-93c15644877c" containerName="revoke-keys"
Nov 01 07:02:11.362689 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.362668    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.405255 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.405226    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.405425 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.405295    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.405425 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.405385    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lksw\" (UniqueName: \"kubernetes.io/projected/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-kube-api-access-4lksw\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.419761 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.419728    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"]
Nov 01 07:02:11.466688 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.466625    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:02:11.466921 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.466697    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:02:11.466921 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.466721    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:02:11.466921 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.466747    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:02:11.506631 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.506560    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.506750 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.506648    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.506750 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.506672    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lksw\" (UniqueName: \"kubernetes.io/projected/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-kube-api-access-4lksw\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.507266 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.507246    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.509246 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.509224    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.567386 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.567354    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lksw\" (UniqueName: \"kubernetes.io/projected/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-kube-api-access-4lksw\") pod \"maas-api-revoke-sub-24c727f1-2gh6r\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:11.675302 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:11.675274    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:12.070594 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:12.070565    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"]
Nov 01 07:02:12.072287 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:02:12.072259    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b0fc50_0c6f_40bb_bb4b_d54eff851a9a.slice/crio-56151c6413ead9363245809c3ba1fb8b324a64432f1c9c1e392871ef02fcae3d WatchSource:0}: Error finding container 56151c6413ead9363245809c3ba1fb8b324a64432f1c9c1e392871ef02fcae3d: Status 404 returned error can't find the container with id 56151c6413ead9363245809c3ba1fb8b324a64432f1c9c1e392871ef02fcae3d
Nov 01 07:02:12.174999 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:12.174961    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r" event={"ID":"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a","Type":"ContainerStarted","Data":"56151c6413ead9363245809c3ba1fb8b324a64432f1c9c1e392871ef02fcae3d"}
Nov 01 07:02:13.181947 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:13.181909    5510 generic.go:356] "Generic (PLEG): container finished" podID="68b0fc50-0c6f-40bb-bb4b-d54eff851a9a" containerID="a2071604407d154b36fc4f935d93af69dbc6dd44e7d27dcd3c6ee5d72f83cbf4" exitCode=0
Nov 01 07:02:13.182327 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:13.181998    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r" event={"ID":"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a","Type":"ContainerDied","Data":"a2071604407d154b36fc4f935d93af69dbc6dd44e7d27dcd3c6ee5d72f83cbf4"}
Nov 01 07:02:14.363957 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.363887    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:14.433412 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.433364    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-xks-ca\") pod \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") "
Nov 01 07:02:14.433640 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.433445    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-openshift-ca\") pod \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") "
Nov 01 07:02:14.433932 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.433890    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-openshift-ca" pod "68b0fc50-0c6f-40bb-bb4b-d54eff851a9a" (UID: "68b0fc50-0c6f-40bb-bb4b-d54eff851a9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:02:14.434139 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.434120    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-kube-api-access-4lksw\" (UniqueName: \"kubernetes.io/projected/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-kube-api-access-4lksw\") pod \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\" (UID: \"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a\") "
Nov 01 07:02:14.435063 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.435043    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:14.436595 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.436309    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-xks-ca" pod "68b0fc50-0c6f-40bb-bb4b-d54eff851a9a" (UID: "68b0fc50-0c6f-40bb-bb4b-d54eff851a9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:02:14.436951 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.436925    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-kube-api-access-4lksw" pod "68b0fc50-0c6f-40bb-bb4b-d54eff851a9a" (UID: "68b0fc50-0c6f-40bb-bb4b-d54eff851a9a"). InnerVolumeSpecName "kube-api-access-4lksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:14.536783 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.536745    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lksw\" (UniqueName: \"kubernetes.io/projected/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-kube-api-access-4lksw\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:14.536783 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:14.536786    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:15.212851 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:15.210962    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r" event={"ID":"68b0fc50-0c6f-40bb-bb4b-d54eff851a9a","Type":"ContainerDied","Data":"56151c6413ead9363245809c3ba1fb8b324a64432f1c9c1e392871ef02fcae3d"}
Nov 01 07:02:15.212851 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:15.211027    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56151c6413ead9363245809c3ba1fb8b324a64432f1c9c1e392871ef02fcae3d"
Nov 01 07:02:15.212851 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:15.211096    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"
Nov 01 07:02:17.545222 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:02:17.545180    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:02:32.111920 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.111883    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"]
Nov 01 07:02:32.112391 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.112284    5510 cpu_manager.go:414] "containerMap: removing container" podUID="68b0fc50-0c6f-40bb-bb4b-d54eff851a9a" containerName="revoke-keys"
Nov 01 07:02:32.112391 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.112295    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b0fc50-0c6f-40bb-bb4b-d54eff851a9a" containerName="revoke-keys"
Nov 01 07:02:32.112391 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.112382    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b0fc50-0c6f-40bb-bb4b-d54eff851a9a" containerName="revoke-keys"
Nov 01 07:02:32.115731 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.115715    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.167658 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.167624    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:02:32.194334 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.194301    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjtv\" (UniqueName: \"kubernetes.io/projected/1bb000b1-b6a3-48bb-9644-562463e24f2c-kube-api-access-jnjtv\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.194469 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.194427    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.194520 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.194479    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.217116 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.217085    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:02:32.217243 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.217161    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:02:32.217332 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.217320    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:02:32.220647 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.220626    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"]
Nov 01 07:02:32.295246 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.295223    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.295363 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.295281    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.295363 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.295325    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjtv\" (UniqueName: \"kubernetes.io/projected/1bb000b1-b6a3-48bb-9644-562463e24f2c-kube-api-access-jnjtv\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.295961 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.295941    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.297717 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.297697    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.356074 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.356041    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjtv\" (UniqueName: \"kubernetes.io/projected/1bb000b1-b6a3-48bb-9644-562463e24f2c-kube-api-access-jnjtv\") pod \"maas-api-revoke-sub-6f103976-rqlwq\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.427959 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.427897    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:32.800108 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:32.800084    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"]
Nov 01 07:02:32.801686 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:02:32.801656    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb000b1_b6a3_48bb_9644_562463e24f2c.slice/crio-4b0b329b786c79f7bbc6b9e482f783fdb3a8b22144444222def83179773a2383 WatchSource:0}: Error finding container 4b0b329b786c79f7bbc6b9e482f783fdb3a8b22144444222def83179773a2383: Status 404 returned error can't find the container with id 4b0b329b786c79f7bbc6b9e482f783fdb3a8b22144444222def83179773a2383
Nov 01 07:02:33.349611 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:33.349518    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq" event={"ID":"1bb000b1-b6a3-48bb-9644-562463e24f2c","Type":"ContainerStarted","Data":"f08f3fcb7e5806eba91fef1bff8668f6a19d748cec8d5cdaa3b3c281942136c7"}
Nov 01 07:02:33.349611 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:33.349570    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq" event={"ID":"1bb000b1-b6a3-48bb-9644-562463e24f2c","Type":"ContainerStarted","Data":"4b0b329b786c79f7bbc6b9e482f783fdb3a8b22144444222def83179773a2383"}
Nov 01 07:02:33.465564 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:33.465515    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq" podStartSLOduration=1.465500816 podStartE2EDuration="1.465500816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:02:33.463964061 +0000 UTC m=+4590.341937472" watchObservedRunningTime="2026-11-01 07:02:33.465500816 +0000 UTC m=+4590.343474225"
Nov 01 07:02:34.356604 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:34.356575    5510 generic.go:356] "Generic (PLEG): container finished" podID="1bb000b1-b6a3-48bb-9644-562463e24f2c" containerID="f08f3fcb7e5806eba91fef1bff8668f6a19d748cec8d5cdaa3b3c281942136c7" exitCode=0
Nov 01 07:02:34.357054 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:34.356653    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq" event={"ID":"1bb000b1-b6a3-48bb-9644-562463e24f2c","Type":"ContainerDied","Data":"f08f3fcb7e5806eba91fef1bff8668f6a19d748cec8d5cdaa3b3c281942136c7"}
Nov 01 07:02:35.492299 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.492276    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:35.627300 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.627220    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-xks-ca\") pod \"1bb000b1-b6a3-48bb-9644-562463e24f2c\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") "
Nov 01 07:02:35.627473 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.627300    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-openshift-ca\") pod \"1bb000b1-b6a3-48bb-9644-562463e24f2c\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") "
Nov 01 07:02:35.627473 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.627385    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb000b1-b6a3-48bb-9644-562463e24f2c-kube-api-access-jnjtv\" (UniqueName: \"kubernetes.io/projected/1bb000b1-b6a3-48bb-9644-562463e24f2c-kube-api-access-jnjtv\") pod \"1bb000b1-b6a3-48bb-9644-562463e24f2c\" (UID: \"1bb000b1-b6a3-48bb-9644-562463e24f2c\") "
Nov 01 07:02:35.627690 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.627655    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-openshift-ca" pod "1bb000b1-b6a3-48bb-9644-562463e24f2c" (UID: "1bb000b1-b6a3-48bb-9644-562463e24f2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:02:35.629662 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.629636    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-xks-ca" pod "1bb000b1-b6a3-48bb-9644-562463e24f2c" (UID: "1bb000b1-b6a3-48bb-9644-562463e24f2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:02:35.629777 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.629655    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb000b1-b6a3-48bb-9644-562463e24f2c-kube-api-access-jnjtv" pod "1bb000b1-b6a3-48bb-9644-562463e24f2c" (UID: "1bb000b1-b6a3-48bb-9644-562463e24f2c"). InnerVolumeSpecName "kube-api-access-jnjtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:35.728247 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.728217    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:35.728395 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.728252    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bb000b1-b6a3-48bb-9644-562463e24f2c-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:35.728395 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:35.728271    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnjtv\" (UniqueName: \"kubernetes.io/projected/1bb000b1-b6a3-48bb-9644-562463e24f2c-kube-api-access-jnjtv\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:36.366905 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:36.366874    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"
Nov 01 07:02:36.367202 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:36.366896    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq" event={"ID":"1bb000b1-b6a3-48bb-9644-562463e24f2c","Type":"ContainerDied","Data":"4b0b329b786c79f7bbc6b9e482f783fdb3a8b22144444222def83179773a2383"}
Nov 01 07:02:36.367335 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:36.367225    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b0b329b786c79f7bbc6b9e482f783fdb3a8b22144444222def83179773a2383"
Nov 01 07:02:42.921404 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.921370    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"]
Nov 01 07:02:42.921871 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.921789    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1bb000b1-b6a3-48bb-9644-562463e24f2c" containerName="revoke-keys"
Nov 01 07:02:42.921871 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.921802    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb000b1-b6a3-48bb-9644-562463e24f2c" containerName="revoke-keys"
Nov 01 07:02:42.921968 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.921895    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb000b1-b6a3-48bb-9644-562463e24f2c" containerName="revoke-keys"
Nov 01 07:02:42.925119 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.925104    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:42.976933 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.976903    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:02:42.996492 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.996462    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kmq\" (UniqueName: \"kubernetes.io/projected/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-kube-api-access-r4kmq\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:42.996641 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.996612    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:42.996706 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:42.996669    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:43.026501 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.026448    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:02:43.026501 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.026462    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:02:43.026501 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.026482    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:02:43.028517 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.028494    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"]
Nov 01 07:02:43.097923 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.097887    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:43.098046 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.097951    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:43.098046 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.097990    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kmq\" (UniqueName: \"kubernetes.io/projected/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-kube-api-access-r4kmq\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:43.098551 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.098530    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:43.100469 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.100443    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:43.158474 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.158440    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kmq\" (UniqueName: \"kubernetes.io/projected/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-kube-api-access-r4kmq\") pod \"maas-api-revoke-sub-5a445fb5-nknh8\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:43.237403 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.237382    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:43.607772 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:43.607724    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"]
Nov 01 07:02:43.610393 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:02:43.610367    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b719ba_bfde_42a8_9e2d_0a15ef0f5e9f.slice/crio-6e30ecb11907cf2c8f64d59c0e1117ad411a12aeff473b29555cc0a4601d5380 WatchSource:0}: Error finding container 6e30ecb11907cf2c8f64d59c0e1117ad411a12aeff473b29555cc0a4601d5380: Status 404 returned error can't find the container with id 6e30ecb11907cf2c8f64d59c0e1117ad411a12aeff473b29555cc0a4601d5380
Nov 01 07:02:44.414356 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:44.414322    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8" event={"ID":"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f","Type":"ContainerStarted","Data":"adf9e9fa26018a173d12fc1a30561b9c77930e63ec4ae0b672f72ced3d706cf7"}
Nov 01 07:02:44.414768 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:44.414374    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8" event={"ID":"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f","Type":"ContainerStarted","Data":"6e30ecb11907cf2c8f64d59c0e1117ad411a12aeff473b29555cc0a4601d5380"}
Nov 01 07:02:44.529232 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:44.529177    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8" podStartSLOduration=2.5291603609999997 podStartE2EDuration="2.529160361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:02:44.528431037 +0000 UTC m=+4601.406404448" watchObservedRunningTime="2026-11-01 07:02:44.529160361 +0000 UTC m=+4601.407133772"
Nov 01 07:02:45.420547 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:45.420509    5510 generic.go:356] "Generic (PLEG): container finished" podID="21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f" containerID="adf9e9fa26018a173d12fc1a30561b9c77930e63ec4ae0b672f72ced3d706cf7" exitCode=0
Nov 01 07:02:45.420983 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:45.420553    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8" event={"ID":"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f","Type":"ContainerDied","Data":"adf9e9fa26018a173d12fc1a30561b9c77930e63ec4ae0b672f72ced3d706cf7"}
Nov 01 07:02:46.698946 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.698880    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:46.767998 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.766233    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-kube-api-access-r4kmq\" (UniqueName: \"kubernetes.io/projected/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-kube-api-access-r4kmq\") pod \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") "
Nov 01 07:02:46.767998 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.766285    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-xks-ca\") pod \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") "
Nov 01 07:02:46.767998 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.766365    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-openshift-ca\") pod \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\" (UID: \"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f\") "
Nov 01 07:02:46.767998 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.767023    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-openshift-ca" pod "21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f" (UID: "21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:02:46.788864 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.781016    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-kube-api-access-r4kmq" pod "21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f" (UID: "21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f"). InnerVolumeSpecName "kube-api-access-r4kmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:46.788864 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.782452    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-xks-ca" pod "21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f" (UID: "21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:02:46.877852 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.870428    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kmq\" (UniqueName: \"kubernetes.io/projected/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-kube-api-access-r4kmq\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:46.881908 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.881879    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:46.893850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:46.889885    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:47.488888 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:47.488816    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8" event={"ID":"21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f","Type":"ContainerDied","Data":"6e30ecb11907cf2c8f64d59c0e1117ad411a12aeff473b29555cc0a4601d5380"}
Nov 01 07:02:47.489164 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:47.489146    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e30ecb11907cf2c8f64d59c0e1117ad411a12aeff473b29555cc0a4601d5380"
Nov 01 07:02:47.489334 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:47.489320    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"
Nov 01 07:02:47.763545 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:02:47.763499    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:02:53.720224 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.720190    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"]
Nov 01 07:02:53.720619 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.720606    5510 cpu_manager.go:414] "containerMap: removing container" podUID="21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f" containerName="revoke-keys"
Nov 01 07:02:53.720671 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.720620    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f" containerName="revoke-keys"
Nov 01 07:02:53.720709 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.720684    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f" containerName="revoke-keys"
Nov 01 07:02:53.723536 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.723517    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:53.775605 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.775582    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:02:53.775777 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.775739    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:02:53.825210 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.825172    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:02:53.825210 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.825187    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:02:53.827448 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.827425    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"]
Nov 01 07:02:53.843134 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.843113    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:53.843252 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.843163    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7s2\" (UniqueName: \"kubernetes.io/projected/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-kube-api-access-4x7s2\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:53.843311 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.843244    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:53.944120 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.944078    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7s2\" (UniqueName: \"kubernetes.io/projected/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-kube-api-access-4x7s2\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:53.944319 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.944145    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:53.944319 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.944261    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:53.944872 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.944812    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:53.946729 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:53.946705    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:54.004841 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:54.004755    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7s2\" (UniqueName: \"kubernetes.io/projected/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-kube-api-access-4x7s2\") pod \"maas-api-revoke-sub-7dafed0a-v2qvs\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:54.037332 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:54.037305    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:54.410038 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:54.410005    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"]
Nov 01 07:02:54.411974 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:02:54.411939    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a6f47e_8aeb_4b2d_8cc4_4cd23c0bb7df.slice/crio-8aaf4d6857c574ef6a986f1a763d52ff2f188c2daa4f92ae1d34da326490c3b7 WatchSource:0}: Error finding container 8aaf4d6857c574ef6a986f1a763d52ff2f188c2daa4f92ae1d34da326490c3b7: Status 404 returned error can't find the container with id 8aaf4d6857c574ef6a986f1a763d52ff2f188c2daa4f92ae1d34da326490c3b7
Nov 01 07:02:54.530377 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:54.530342    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs" event={"ID":"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df","Type":"ContainerStarted","Data":"8aaf4d6857c574ef6a986f1a763d52ff2f188c2daa4f92ae1d34da326490c3b7"}
Nov 01 07:02:55.537520 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:55.537487    5510 generic.go:356] "Generic (PLEG): container finished" podID="23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df" containerID="ee41f2ef93e9222edea6f0d6bff23bd8e639c2c0bc7804127966f830cdbe2e7d" exitCode=0
Nov 01 07:02:55.537939 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:55.537547    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs" event={"ID":"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df","Type":"ContainerDied","Data":"ee41f2ef93e9222edea6f0d6bff23bd8e639c2c0bc7804127966f830cdbe2e7d"}
Nov 01 07:02:56.708859 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.708809    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:56.873265 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.873170    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-xks-ca\") pod \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") "
Nov 01 07:02:56.873440 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.873290    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-openshift-ca\") pod \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") "
Nov 01 07:02:56.873440 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.873357    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-kube-api-access-4x7s2\" (UniqueName: \"kubernetes.io/projected/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-kube-api-access-4x7s2\") pod \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\" (UID: \"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df\") "
Nov 01 07:02:56.874275 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.874244    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-openshift-ca" pod "23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df" (UID: "23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:02:56.876960 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.876936    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-xks-ca" pod "23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df" (UID: "23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:02:56.880448 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.880413    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-kube-api-access-4x7s2" pod "23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df" (UID: "23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df"). InnerVolumeSpecName "kube-api-access-4x7s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:56.975391 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.975350    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:56.975391 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.975387    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7s2\" (UniqueName: \"kubernetes.io/projected/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-kube-api-access-4x7s2\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:56.975644 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:56.975409    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:57.568903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:57.563208    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs" event={"ID":"23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df","Type":"ContainerDied","Data":"8aaf4d6857c574ef6a986f1a763d52ff2f188c2daa4f92ae1d34da326490c3b7"}
Nov 01 07:02:57.568903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:57.563235    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"
Nov 01 07:02:57.568903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:02:57.563265    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aaf4d6857c574ef6a986f1a763d52ff2f188c2daa4f92ae1d34da326490c3b7"
Nov 01 07:03:04.524349 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.524311    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"]
Nov 01 07:03:04.525004 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.524957    5510 cpu_manager.go:414] "containerMap: removing container" podUID="23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df" containerName="revoke-keys"
Nov 01 07:03:04.525004 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.524976    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df" containerName="revoke-keys"
Nov 01 07:03:04.525135 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.525090    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df" containerName="revoke-keys"
Nov 01 07:03:04.528604 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.528585    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.583206 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.583159    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:03:04.596680 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.596652    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr6bp\" (UniqueName: \"kubernetes.io/projected/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-kube-api-access-gr6bp\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.596810 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.596711    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.596810 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.596783    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.629931 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.629903    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:03:04.630115 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.630097    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:03:04.630187 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.630164    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:03:04.634141 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.634116    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"]
Nov 01 07:03:04.698193 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.698158    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.698351 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.698208    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.698351 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.698314    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6bp\" (UniqueName: \"kubernetes.io/projected/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-kube-api-access-gr6bp\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.698899 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.698879    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.700724 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.700708    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.759000 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.758969    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6bp\" (UniqueName: \"kubernetes.io/projected/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-kube-api-access-gr6bp\") pod \"maas-api-revoke-sub-384805a7-hpzfd\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:04.842290 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:04.842230    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:05.220349 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:05.220320    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"]
Nov 01 07:03:05.222348 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:03:05.222321    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403604c1_aefb_48e3_bdd5_a1a4658bc1f2.slice/crio-bfcc6da1b9ab40349e9b15dbed484243c54da77210cbb5de6c18ab8104b8f9bf WatchSource:0}: Error finding container bfcc6da1b9ab40349e9b15dbed484243c54da77210cbb5de6c18ab8104b8f9bf: Status 404 returned error can't find the container with id bfcc6da1b9ab40349e9b15dbed484243c54da77210cbb5de6c18ab8104b8f9bf
Nov 01 07:03:05.620404 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:05.620356    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd" event={"ID":"403604c1-aefb-48e3-bdd5-a1a4658bc1f2","Type":"ContainerStarted","Data":"c4ee8648b69452b557c64821d2a089966655339c94bc83a9bd93f58ceed39228"}
Nov 01 07:03:05.620867 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:05.620413    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd" event={"ID":"403604c1-aefb-48e3-bdd5-a1a4658bc1f2","Type":"ContainerStarted","Data":"bfcc6da1b9ab40349e9b15dbed484243c54da77210cbb5de6c18ab8104b8f9bf"}
Nov 01 07:03:05.736518 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:05.736423    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd" podStartSLOduration=1.736410449 podStartE2EDuration="1.736410449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:05.734538544 +0000 UTC m=+4622.612511954" watchObservedRunningTime="2026-11-01 07:03:05.736410449 +0000 UTC m=+4622.614383892"
Nov 01 07:03:06.627006 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:06.626968    5510 generic.go:356] "Generic (PLEG): container finished" podID="403604c1-aefb-48e3-bdd5-a1a4658bc1f2" containerID="c4ee8648b69452b557c64821d2a089966655339c94bc83a9bd93f58ceed39228" exitCode=0
Nov 01 07:03:06.627427 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:06.627056    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd" event={"ID":"403604c1-aefb-48e3-bdd5-a1a4658bc1f2","Type":"ContainerDied","Data":"c4ee8648b69452b557c64821d2a089966655339c94bc83a9bd93f58ceed39228"}
Nov 01 07:03:08.139351 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.139002    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:08.274858 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.273815    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-xks-ca\") pod \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") "
Nov 01 07:03:08.274858 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.273933    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-kube-api-access-gr6bp\" (UniqueName: \"kubernetes.io/projected/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-kube-api-access-gr6bp\") pod \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") "
Nov 01 07:03:08.274858 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.274017    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-openshift-ca\") pod \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\" (UID: \"403604c1-aefb-48e3-bdd5-a1a4658bc1f2\") "
Nov 01 07:03:08.284057 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.284030    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-openshift-ca" pod "403604c1-aefb-48e3-bdd5-a1a4658bc1f2" (UID: "403604c1-aefb-48e3-bdd5-a1a4658bc1f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:08.285522 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.285495    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-kube-api-access-gr6bp" pod "403604c1-aefb-48e3-bdd5-a1a4658bc1f2" (UID: "403604c1-aefb-48e3-bdd5-a1a4658bc1f2"). InnerVolumeSpecName "kube-api-access-gr6bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:08.314738 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.314705    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-xks-ca" pod "403604c1-aefb-48e3-bdd5-a1a4658bc1f2" (UID: "403604c1-aefb-48e3-bdd5-a1a4658bc1f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:08.385912 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.378456    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:08.389850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.386145    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:08.390046 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.390025    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr6bp\" (UniqueName: \"kubernetes.io/projected/403604c1-aefb-48e3-bdd5-a1a4658bc1f2-kube-api-access-gr6bp\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:08.651420 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.651325    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd" event={"ID":"403604c1-aefb-48e3-bdd5-a1a4658bc1f2","Type":"ContainerDied","Data":"bfcc6da1b9ab40349e9b15dbed484243c54da77210cbb5de6c18ab8104b8f9bf"}
Nov 01 07:03:08.651675 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.651657    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfcc6da1b9ab40349e9b15dbed484243c54da77210cbb5de6c18ab8104b8f9bf"
Nov 01 07:03:08.651879 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:08.651865    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"
Nov 01 07:03:17.544416 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:03:17.544381    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:03:17.676488 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.676447    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:03:17.848373 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.848293    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"]
Nov 01 07:03:17.848722 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.848710    5510 cpu_manager.go:414] "containerMap: removing container" podUID="403604c1-aefb-48e3-bdd5-a1a4658bc1f2" containerName="revoke-keys"
Nov 01 07:03:17.848770 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.848723    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403604c1-aefb-48e3-bdd5-a1a4658bc1f2" containerName="revoke-keys"
Nov 01 07:03:17.848812 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.848787    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="403604c1-aefb-48e3-bdd5-a1a4658bc1f2" containerName="revoke-keys"
Nov 01 07:03:17.852010 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.851995    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:17.892394 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.892370    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:17.892552 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.892407    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qn2\" (UniqueName: \"kubernetes.io/projected/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-kube-api-access-t2qn2\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:17.892552 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.892439    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:17.953283 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.953226    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:03:17.953283 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.953256    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:03:17.953283 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.953260    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:03:17.953600 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.953302    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:03:17.955508 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.955483    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"]
Nov 01 07:03:17.993528 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.993503    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:17.993642 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.993538    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qn2\" (UniqueName: \"kubernetes.io/projected/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-kube-api-access-t2qn2\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:17.993642 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.993565    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:17.994199 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.994181    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:17.996016 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:17.995999    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:18.055663 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:18.055634    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qn2\" (UniqueName: \"kubernetes.io/projected/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-kube-api-access-t2qn2\") pod \"maas-api-revoke-sub-300feb70-k7nrn\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:18.165352 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:18.165247    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:18.679855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:18.676717    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"]
Nov 01 07:03:18.725432 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:03:18.725398    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485f113d_f8a9_4c10_bb5d_520f0a6b02cc.slice/crio-76dba7bc361daf3c28158e179675eeb82441413f1831e318a23e621f9dbc052f WatchSource:0}: Error finding container 76dba7bc361daf3c28158e179675eeb82441413f1831e318a23e621f9dbc052f: Status 404 returned error can't find the container with id 76dba7bc361daf3c28158e179675eeb82441413f1831e318a23e621f9dbc052f
Nov 01 07:03:19.464854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:19.462917    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:03:19.464854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:19.462975    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:03:19.564909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:19.563973    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:03:19.564909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:19.564035    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:03:19.757482 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:19.757448    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn" event={"ID":"485f113d-f8a9-4c10-bb5d-520f0a6b02cc","Type":"ContainerStarted","Data":"8f4361661ec032488f72e0b398d1292630257a141d4b86563e137848e5f5dd13"}
Nov 01 07:03:19.758152 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:19.758114    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn" event={"ID":"485f113d-f8a9-4c10-bb5d-520f0a6b02cc","Type":"ContainerStarted","Data":"76dba7bc361daf3c28158e179675eeb82441413f1831e318a23e621f9dbc052f"}
Nov 01 07:03:19.889652 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:19.889509    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn" podStartSLOduration=2.889487882 podStartE2EDuration="2.889487882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:19.888346963 +0000 UTC m=+4636.766320376" watchObservedRunningTime="2026-11-01 07:03:19.889487882 +0000 UTC m=+4636.767461293"
Nov 01 07:03:20.763881 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:20.763851    5510 generic.go:356] "Generic (PLEG): container finished" podID="485f113d-f8a9-4c10-bb5d-520f0a6b02cc" containerID="8f4361661ec032488f72e0b398d1292630257a141d4b86563e137848e5f5dd13" exitCode=0
Nov 01 07:03:20.764302 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:20.763889    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn" event={"ID":"485f113d-f8a9-4c10-bb5d-520f0a6b02cc","Type":"ContainerDied","Data":"8f4361661ec032488f72e0b398d1292630257a141d4b86563e137848e5f5dd13"}
Nov 01 07:03:21.912757 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:21.912728    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:21.970922 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:21.970900    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-xks-ca\") pod \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") "
Nov 01 07:03:21.971059 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:21.970965    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-kube-api-access-t2qn2\" (UniqueName: \"kubernetes.io/projected/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-kube-api-access-t2qn2\") pod \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") "
Nov 01 07:03:21.971059 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:21.971045    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-openshift-ca\") pod \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\" (UID: \"485f113d-f8a9-4c10-bb5d-520f0a6b02cc\") "
Nov 01 07:03:21.971396 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:21.971372    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-openshift-ca" pod "485f113d-f8a9-4c10-bb5d-520f0a6b02cc" (UID: "485f113d-f8a9-4c10-bb5d-520f0a6b02cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:21.972978 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:21.972957    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-xks-ca" pod "485f113d-f8a9-4c10-bb5d-520f0a6b02cc" (UID: "485f113d-f8a9-4c10-bb5d-520f0a6b02cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:21.973094 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:21.973070    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-kube-api-access-t2qn2" pod "485f113d-f8a9-4c10-bb5d-520f0a6b02cc" (UID: "485f113d-f8a9-4c10-bb5d-520f0a6b02cc"). InnerVolumeSpecName "kube-api-access-t2qn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:22.072633 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:22.072557    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qn2\" (UniqueName: \"kubernetes.io/projected/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-kube-api-access-t2qn2\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:22.072633 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:22.072582    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:22.072633 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:22.072592    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485f113d-f8a9-4c10-bb5d-520f0a6b02cc-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:22.775638 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:22.775608    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"
Nov 01 07:03:22.775849 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:22.775643    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn" event={"ID":"485f113d-f8a9-4c10-bb5d-520f0a6b02cc","Type":"ContainerDied","Data":"76dba7bc361daf3c28158e179675eeb82441413f1831e318a23e621f9dbc052f"}
Nov 01 07:03:22.775849 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:22.775691    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76dba7bc361daf3c28158e179675eeb82441413f1831e318a23e621f9dbc052f"
Nov 01 07:03:28.410262 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:28.410225    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:03:29.586891 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:29.586857    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"]
Nov 01 07:03:29.960097 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:29.960067    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"]
Nov 01 07:03:29.960535 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:29.960519    5510 cpu_manager.go:414] "containerMap: removing container" podUID="485f113d-f8a9-4c10-bb5d-520f0a6b02cc" containerName="revoke-keys"
Nov 01 07:03:29.960597 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:29.960538    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485f113d-f8a9-4c10-bb5d-520f0a6b02cc" containerName="revoke-keys"
Nov 01 07:03:29.960641 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:29.960619    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="485f113d-f8a9-4c10-bb5d-520f0a6b02cc" containerName="revoke-keys"
Nov 01 07:03:29.963945 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:29.963928    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.023926 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.019882    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:03:30.023926 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.020000    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h4cjj\""
Nov 01 07:03:30.023926 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.020050    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:03:30.023926 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.020107    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:03:30.025355 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.025325    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"]
Nov 01 07:03:30.052153 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.052120    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.052315 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.052201    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzlz\" (UniqueName: \"kubernetes.io/projected/abdda2b1-98f4-4ef3-a296-116a87e0e43d-kube-api-access-5zzlz\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.052361 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.052329    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.153202 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.153166    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.153399 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.153283    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzlz\" (UniqueName: \"kubernetes.io/projected/abdda2b1-98f4-4ef3-a296-116a87e0e43d-kube-api-access-5zzlz\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.153399 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.153328    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.153809 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.153789    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.155809 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.155786    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.214183 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.214120    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzlz\" (UniqueName: \"kubernetes.io/projected/abdda2b1-98f4-4ef3-a296-116a87e0e43d-kube-api-access-5zzlz\") pod \"maas-api-revoke-sub-429d9d8e-p5btm\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.276764 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.276728    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:30.652047 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.652015    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"]
Nov 01 07:03:30.653690 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:03:30.653662    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdda2b1_98f4_4ef3_a296_116a87e0e43d.slice/crio-fe99b652135aa4de86e19b9feeb170e09c2efb9029037b8502539132e9118598 WatchSource:0}: Error finding container fe99b652135aa4de86e19b9feeb170e09c2efb9029037b8502539132e9118598: Status 404 returned error can't find the container with id fe99b652135aa4de86e19b9feeb170e09c2efb9029037b8502539132e9118598
Nov 01 07:03:30.820728 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:30.820685    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm" event={"ID":"abdda2b1-98f4-4ef3-a296-116a87e0e43d","Type":"ContainerStarted","Data":"fe99b652135aa4de86e19b9feeb170e09c2efb9029037b8502539132e9118598"}
Nov 01 07:03:31.827941 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:31.827856    5510 generic.go:356] "Generic (PLEG): container finished" podID="abdda2b1-98f4-4ef3-a296-116a87e0e43d" containerID="b003fdfdd7dd19f169e6e2697d3ad22d90953b2c9ebf031b5951d386251e02cb" exitCode=0
Nov 01 07:03:31.828347 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:31.827932    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm" event={"ID":"abdda2b1-98f4-4ef3-a296-116a87e0e43d","Type":"ContainerDied","Data":"b003fdfdd7dd19f169e6e2697d3ad22d90953b2c9ebf031b5951d386251e02cb"}
Nov 01 07:03:32.962392 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:32.962365    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:33.083780 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.083701    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdda2b1-98f4-4ef3-a296-116a87e0e43d-kube-api-access-5zzlz\" (UniqueName: \"kubernetes.io/projected/abdda2b1-98f4-4ef3-a296-116a87e0e43d-kube-api-access-5zzlz\") pod \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") "
Nov 01 07:03:33.083780 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.083744    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-openshift-ca\") pod \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") "
Nov 01 07:03:33.084014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.083820    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-xks-ca\") pod \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\" (UID: \"abdda2b1-98f4-4ef3-a296-116a87e0e43d\") "
Nov 01 07:03:33.084199 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.084178    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-openshift-ca" pod "abdda2b1-98f4-4ef3-a296-116a87e0e43d" (UID: "abdda2b1-98f4-4ef3-a296-116a87e0e43d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:33.086089 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.086063    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdda2b1-98f4-4ef3-a296-116a87e0e43d-kube-api-access-5zzlz" pod "abdda2b1-98f4-4ef3-a296-116a87e0e43d" (UID: "abdda2b1-98f4-4ef3-a296-116a87e0e43d"). InnerVolumeSpecName "kube-api-access-5zzlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:33.086318 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.086301    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-xks-ca" pod "abdda2b1-98f4-4ef3-a296-116a87e0e43d" (UID: "abdda2b1-98f4-4ef3-a296-116a87e0e43d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:33.184668 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.184635    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zzlz\" (UniqueName: \"kubernetes.io/projected/abdda2b1-98f4-4ef3-a296-116a87e0e43d-kube-api-access-5zzlz\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:33.184668 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.184664    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:33.184668 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.184674    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abdda2b1-98f4-4ef3-a296-116a87e0e43d-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:33.840243 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.840213    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"
Nov 01 07:03:33.840243 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.840239    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm" event={"ID":"abdda2b1-98f4-4ef3-a296-116a87e0e43d","Type":"ContainerDied","Data":"fe99b652135aa4de86e19b9feeb170e09c2efb9029037b8502539132e9118598"}
Nov 01 07:03:33.840473 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:33.840289    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe99b652135aa4de86e19b9feeb170e09c2efb9029037b8502539132e9118598"
Nov 01 07:03:34.588248 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:34.588207    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2" podUID="baa242a0-7d0f-423c-8278-cecdd18222e8" containerName="payload-pre-processing" containerID="cri-o://4d446cfa7eb34cea2488506f7d46e5b9a3ecf09c638151aab70cfc938b9049ab" gracePeriod=25
Nov 01 07:03:34.736423 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:03:34.736393    5510 logging.go:55] [core] [Channel #14756 SubChannel #14757]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.152:50052", ServerName: "10.134.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.152:50052: connect: connection refused"
Nov 01 07:03:34.849203 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:34.849173    5510 generic.go:356] "Generic (PLEG): container finished" podID="baa242a0-7d0f-423c-8278-cecdd18222e8" containerID="4d446cfa7eb34cea2488506f7d46e5b9a3ecf09c638151aab70cfc938b9049ab" exitCode=0
Nov 01 07:03:34.849378 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:34.849218    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2" event={"ID":"baa242a0-7d0f-423c-8278-cecdd18222e8","Type":"ContainerDied","Data":"4d446cfa7eb34cea2488506f7d46e5b9a3ecf09c638151aab70cfc938b9049ab"}
Nov 01 07:03:34.849378 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:34.849296    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2" event={"ID":"baa242a0-7d0f-423c-8278-cecdd18222e8","Type":"ContainerDied","Data":"7ed671aecd1b4dd0203937ae90e593b632f25c9181f93126ab74f60c92285a75"}
Nov 01 07:03:34.849378 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:34.849320    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ed671aecd1b4dd0203937ae90e593b632f25c9181f93126ab74f60c92285a75"
Nov 01 07:03:34.859230 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:34.859198    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 07:03:35.003094 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:35.003055    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa242a0-7d0f-423c-8278-cecdd18222e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa242a0-7d0f-423c-8278-cecdd18222e8-plugins-config-volume\") pod \"baa242a0-7d0f-423c-8278-cecdd18222e8\" (UID: \"baa242a0-7d0f-423c-8278-cecdd18222e8\") "
Nov 01 07:03:35.003254 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:35.003186    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa242a0-7d0f-423c-8278-cecdd18222e8-kube-api-access-nvz8h\" (UniqueName: \"kubernetes.io/projected/baa242a0-7d0f-423c-8278-cecdd18222e8-kube-api-access-nvz8h\") pod \"baa242a0-7d0f-423c-8278-cecdd18222e8\" (UID: \"baa242a0-7d0f-423c-8278-cecdd18222e8\") "
Nov 01 07:03:35.003441 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:35.003417    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa242a0-7d0f-423c-8278-cecdd18222e8-plugins-config-volume" pod "baa242a0-7d0f-423c-8278-cecdd18222e8" (UID: "baa242a0-7d0f-423c-8278-cecdd18222e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:35.003558 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:35.003537    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa242a0-7d0f-423c-8278-cecdd18222e8-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:35.005408 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:35.005379    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa242a0-7d0f-423c-8278-cecdd18222e8-kube-api-access-nvz8h" pod "baa242a0-7d0f-423c-8278-cecdd18222e8" (UID: "baa242a0-7d0f-423c-8278-cecdd18222e8"). InnerVolumeSpecName "kube-api-access-nvz8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:35.104790 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:35.104708    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvz8h\" (UniqueName: \"kubernetes.io/projected/baa242a0-7d0f-423c-8278-cecdd18222e8-kube-api-access-nvz8h\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:35.737067 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:35.737027    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2" podUID="baa242a0-7d0f-423c-8278-cecdd18222e8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.152:50052\" within 1s: context deadline exceeded"
Nov 01 07:03:35.737466 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:03:35.737124    5510 logging.go:55] [core] [Channel #14756 SubChannel #14757]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.152:50052", ServerName: "10.134.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.152:50052: operation was canceled"
Nov 01 07:03:35.854697 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:35.854667    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"
Nov 01 07:03:36.017130 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:36.017037    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"]
Nov 01 07:03:36.066336 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:36.066304    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-045f69-588fdb899f-bdmw2"]
Nov 01 07:03:37.234535 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:37.234493    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa242a0-7d0f-423c-8278-cecdd18222e8" path="/var/lib/kubelet/pods/baa242a0-7d0f-423c-8278-cecdd18222e8/volumes"
Nov 01 07:03:43.461795 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:43.461759    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:03:44.071957 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.071923    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"]
Nov 01 07:03:44.072511 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.072492    5510 cpu_manager.go:414] "containerMap: removing container" podUID="baa242a0-7d0f-423c-8278-cecdd18222e8" containerName="payload-pre-processing"
Nov 01 07:03:44.072610 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.072515    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa242a0-7d0f-423c-8278-cecdd18222e8" containerName="payload-pre-processing"
Nov 01 07:03:44.072610 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.072535    5510 cpu_manager.go:414] "containerMap: removing container" podUID="abdda2b1-98f4-4ef3-a296-116a87e0e43d" containerName="revoke-keys"
Nov 01 07:03:44.072610 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.072543    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdda2b1-98f4-4ef3-a296-116a87e0e43d" containerName="revoke-keys"
Nov 01 07:03:44.072779 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.072659    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa242a0-7d0f-423c-8278-cecdd18222e8" containerName="payload-pre-processing"
Nov 01 07:03:44.072779 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.072672    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdda2b1-98f4-4ef3-a296-116a87e0e43d" containerName="revoke-keys"
Nov 01 07:03:44.076405 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.076379    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.086927 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.086897    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-data\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.087087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.086957    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-podinfo\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.087087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.087068    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.087206 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.087123    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-credential-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.087206 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.087159    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-certs\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.087332 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.087210    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-envoy\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.087332 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.087241    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm752\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-kube-api-access-rm752\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.087332 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.087280    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istiod-ca-cert\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.087332 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.087319    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-token\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.129853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.129805    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-main-045f69-openshift-default-dockercfg-4slwf\""
Nov 01 07:03:44.181702 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.181668    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"]
Nov 01 07:03:44.187906 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.187881    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-certs\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188027 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.187931    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-envoy\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188027 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.187961    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm752\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-kube-api-access-rm752\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188027 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.187997    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istiod-ca-cert\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188156 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188132    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-token\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188264 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188249    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-data\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188315 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188299    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-podinfo\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188357 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188313    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-certs\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188357 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188342    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188472 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188389    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-credential-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188629 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188604    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-data\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188723 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188696    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188723 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188704    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istiod-ca-cert\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.188723 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.188714    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-credential-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.190504 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.190481    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-envoy\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.190691 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.190674    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-podinfo\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.250445 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.250411    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-token\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.250702 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.250681    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm752\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-kube-api-access-rm752\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-rs27n\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.392056 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.391978    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:44.817612 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.817594    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"]
Nov 01 07:03:44.820095 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:03:44.820067    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63aabae8_7351_4b1a_a3c7_6b5f0e7c3d61.slice/crio-59a3bf026f0195a69855b8088cb24e53f061f32e0991420697ca8af461c1b5c5 WatchSource:0}: Error finding container 59a3bf026f0195a69855b8088cb24e53f061f32e0991420697ca8af461c1b5c5: Status 404 returned error can't find the container with id 59a3bf026f0195a69855b8088cb24e53f061f32e0991420697ca8af461c1b5c5
Nov 01 07:03:44.822244 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.822210    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:03:44.822341 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.822271    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:03:44.822341 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.822299    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:03:44.906181 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.906151    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" event={"ID":"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61","Type":"ContainerStarted","Data":"d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a"}
Nov 01 07:03:44.906317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:44.906202    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" event={"ID":"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61","Type":"ContainerStarted","Data":"59a3bf026f0195a69855b8088cb24e53f061f32e0991420697ca8af461c1b5c5"}
Nov 01 07:03:45.027561 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:45.027508    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podStartSLOduration=1.027493824 podStartE2EDuration="1.027493824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:45.024861237 +0000 UTC m=+4661.902834647" watchObservedRunningTime="2026-11-01 07:03:45.027493824 +0000 UTC m=+4661.905467233"
Nov 01 07:03:45.394068 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:45.394019    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:45.395800 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:45.395773    5510 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:03:45.395956 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:45.395836    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 07:03:46.394251 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:46.394216    5510 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:03:46.394642 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:46.394275    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 07:03:47.394218 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:47.394182    5510 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:03:47.394436 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:47.394239    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 07:03:47.544201 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:03:47.544156    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:03:48.394184 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:48.394141    5510 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:03:48.394498 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:48.394198    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 07:03:49.394290 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:49.394249    5510 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:03:49.394521 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:49.394315    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 07:03:50.393854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:50.392868    5510 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:03:50.393854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:50.392977    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 07:03:51.805667 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:51.805634    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:51.806126 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:51.806112    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:51.806911 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:51.806892    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:03:53.796424 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:03:53.796393    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:04:02.563664 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:02.563628    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"]
Nov 01 07:04:02.564112 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:02.563876    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy" containerID="cri-o://d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a" gracePeriod=30
Nov 01 07:04:04.841210 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:04.841176    5510 scope.go:122] "RemoveContainer" containerID="4d446cfa7eb34cea2488506f7d46e5b9a3ecf09c638151aab70cfc938b9049ab"
Nov 01 07:04:04.850614 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:04.850591    5510 scope.go:122] "RemoveContainer" containerID="b55169b407d538dcde483a1ac88398ff95fbe75fe357d001ed2c509b57e4051f"
Nov 01 07:04:06.808439 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:06.808404    5510 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:04:06.808920 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:06.808475    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:04:07.813646 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.813623    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:04:07.919984 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.919904    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-socket\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920141 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.919984    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-podinfo\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920141 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920024    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-envoy\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920141 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920053    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-kube-api-access-rm752\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-kube-api-access-rm752\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920313 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920169    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-certs\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920313 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920209    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-socket" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.920313 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920243    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-token\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-token\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920313 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920283    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-data\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920512 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920319    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-credential-socket\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920512 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920358    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istiod-ca-cert\") pod \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\" (UID: \"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61\") "
Nov 01 07:04:07.920512 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920419    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-certs" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.920663 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920575    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-credential-socket" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.920741 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920722    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.920801 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920749    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-credential-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.920801 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920764    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-workload-socket\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.920920 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920810    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istiod-ca-cert" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:04:07.921010 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.920985    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-data" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.922359 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.922341    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-podinfo" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:04:07.922619 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.922605    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-kube-api-access-rm752" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "kube-api-access-rm752". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:04:07.922925 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.922905    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-token" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:04:07.922925 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:07.922913    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-envoy" pod "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" (UID: "63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:08.022303 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.022276    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-podinfo\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:08.022303 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.022300    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-envoy\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:08.022486 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.022310    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm752\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-kube-api-access-rm752\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:08.022486 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.022320    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-token\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:08.022486 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.022331    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istio-data\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:08.022486 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.022338    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61-istiod-ca-cert\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:08.031385 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.031350    5510 generic.go:356] "Generic (PLEG): container finished" podID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerID="d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a" exitCode=0
Nov 01 07:04:08.031514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.031393    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" event={"ID":"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61","Type":"ContainerDied","Data":"d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a"}
Nov 01 07:04:08.031514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.031405    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"
Nov 01 07:04:08.031514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.031445    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n" event={"ID":"63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61","Type":"ContainerDied","Data":"59a3bf026f0195a69855b8088cb24e53f061f32e0991420697ca8af461c1b5c5"}
Nov 01 07:04:08.031514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.031474    5510 scope.go:122] "RemoveContainer" containerID="d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a"
Nov 01 07:04:08.043620 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.043607    5510 scope.go:122] "RemoveContainer" containerID="d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a"
Nov 01 07:04:08.043891 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:04:08.043873    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a\": container with ID starting with d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a not found: ID does not exist" containerID="d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a"
Nov 01 07:04:08.043944 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.043900    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a"} err="failed to get container status \"d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a\": rpc error: code = NotFound desc = could not find container \"d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a\": container with ID starting with d6248c7bbdee68bb5247595fb30a738d595859eb58db13931938bead55999c1a not found: ID does not exist"
Nov 01 07:04:08.195445 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.195362    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"]
Nov 01 07:04:08.201719 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:08.201689    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-rs27n"]
Nov 01 07:04:09.234555 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:09.234516    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" path="/var/lib/kubelet/pods/63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61/volumes"
Nov 01 07:04:17.548670 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:04:17.548573    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:04:34.834530 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.834498    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"]
Nov 01 07:04:34.835164 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.835145    5510 cpu_manager.go:414] "containerMap: removing container" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy"
Nov 01 07:04:34.835241 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.835163    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy"
Nov 01 07:04:34.835300 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.835290    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="63aabae8-7351-4b1a-a3c7-6b5f0e7c3d61" containerName="istio-proxy"
Nov 01 07:04:34.838576 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.838556    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:34.869246 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.869223    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dea24-5bb0-4858-bfda-b74cde8ed32d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r\" (UID: \"128dea24-5bb0-4858-bfda-b74cde8ed32d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:34.869375 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.869361    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8x4\" (UniqueName: \"kubernetes.io/projected/128dea24-5bb0-4858-bfda-b74cde8ed32d-kube-api-access-sh8x4\") pod \"payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r\" (UID: \"128dea24-5bb0-4858-bfda-b74cde8ed32d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:34.890474 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.890452    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9af193c3\""
Nov 01 07:04:34.942852 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.940072    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9af193c3-dockercfg-r4q58\""
Nov 01 07:04:34.944338 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.944309    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"]
Nov 01 07:04:34.969933 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.969891    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8x4\" (UniqueName: \"kubernetes.io/projected/128dea24-5bb0-4858-bfda-b74cde8ed32d-kube-api-access-sh8x4\") pod \"payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r\" (UID: \"128dea24-5bb0-4858-bfda-b74cde8ed32d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:34.970149 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.969963    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dea24-5bb0-4858-bfda-b74cde8ed32d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r\" (UID: \"128dea24-5bb0-4858-bfda-b74cde8ed32d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:34.971050 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:34.971025    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dea24-5bb0-4858-bfda-b74cde8ed32d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r\" (UID: \"128dea24-5bb0-4858-bfda-b74cde8ed32d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:35.031512 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:35.031488    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8x4\" (UniqueName: \"kubernetes.io/projected/128dea24-5bb0-4858-bfda-b74cde8ed32d-kube-api-access-sh8x4\") pod \"payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r\" (UID: \"128dea24-5bb0-4858-bfda-b74cde8ed32d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:35.152365 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:35.152293    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:35.525157 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:35.525130    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"]
Nov 01 07:04:35.526896 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:04:35.526871    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128dea24_5bb0_4858_bfda_b74cde8ed32d.slice/crio-8a76881ccf630a9d67c2a01399f372490200bf4ea93231d87f61e89ae489f5f4 WatchSource:0}: Error finding container 8a76881ccf630a9d67c2a01399f372490200bf4ea93231d87f61e89ae489f5f4: Status 404 returned error can't find the container with id 8a76881ccf630a9d67c2a01399f372490200bf4ea93231d87f61e89ae489f5f4
Nov 01 07:04:36.176401 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:36.176363    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r" event={"ID":"128dea24-5bb0-4858-bfda-b74cde8ed32d","Type":"ContainerStarted","Data":"bdd2584d56f01193e1c8e72506c24312da78b5bc5ec995219bbbd89be123c42a"}
Nov 01 07:04:36.176809 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:36.176439    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r" event={"ID":"128dea24-5bb0-4858-bfda-b74cde8ed32d","Type":"ContainerStarted","Data":"8a76881ccf630a9d67c2a01399f372490200bf4ea93231d87f61e89ae489f5f4"}
Nov 01 07:04:36.176809 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:36.176498    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:36.298937 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:36.298874    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r" podStartSLOduration=2.298847978 podStartE2EDuration="2.298847978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:04:36.297598079 +0000 UTC m=+4713.175571488" watchObservedRunningTime="2026-11-01 07:04:36.298847978 +0000 UTC m=+4713.176821400"
Nov 01 07:04:37.315007 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.314966    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"]
Nov 01 07:04:37.318821 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.318797    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.370969 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.370942    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9af193c3-serving-cert\""
Nov 01 07:04:37.397484 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.397440    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-metrics-certs\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.397610 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.397497    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shd7k\" (UniqueName: \"kubernetes.io/projected/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-kube-api-access-shd7k\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.397664 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.397628    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-openshift-service-ca\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.397710 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.397667    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.420340 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.420309    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:04:37.420474 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.420403    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 07:04:37.420474 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.420416    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:04:37.420474 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.420453    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rqph5\""
Nov 01 07:04:37.422791 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.422768    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"]
Nov 01 07:04:37.498545 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.498516    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-openshift-service-ca\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.498748 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.498560    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.498748 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.498664    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-metrics-certs\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.498748 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.498688    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shd7k\" (UniqueName: \"kubernetes.io/projected/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-kube-api-access-shd7k\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.498748 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:04:37.498735    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-serving-cert: secret "maas-api-e2e-ait-9af193c3-serving-cert" not found
Nov 01 07:04:37.499007 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:04:37.498813    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls podName:9135ebe4-7ef2-4383-8a5a-3df3b11070eb nodeName:}" failed. No retries permitted until 2026-11-01 07:04:37.998792042 +0000 UTC m=+4714.876765431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls") pod "maas-api-e2e-ait-9af193c3-848bbb48-vdwx7" (UID: "9135ebe4-7ef2-4383-8a5a-3df3b11070eb") : secret "maas-api-e2e-ait-9af193c3-serving-cert" not found
Nov 01 07:04:37.499200 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.499179    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-openshift-service-ca\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.501182 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.501165    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-metrics-certs\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:37.559503 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:37.559477    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shd7k\" (UniqueName: \"kubernetes.io/projected/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-kube-api-access-shd7k\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:38.003125 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:38.003091    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:38.005620 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:38.005596    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls\") pod \"maas-api-e2e-ait-9af193c3-848bbb48-vdwx7\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:38.231618 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:38.231583    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:38.604960 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:38.604937    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"]
Nov 01 07:04:38.606344 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:04:38.606314    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9135ebe4_7ef2_4383_8a5a_3df3b11070eb.slice/crio-c6417cccffa1d237c27e45c9decd77648145e4ae8d4d047ff180a26b00310a49 WatchSource:0}: Error finding container c6417cccffa1d237c27e45c9decd77648145e4ae8d4d047ff180a26b00310a49: Status 404 returned error can't find the container with id c6417cccffa1d237c27e45c9decd77648145e4ae8d4d047ff180a26b00310a49
Nov 01 07:04:38.607692 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:38.607675    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 07:04:39.194123 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:39.194086    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7" event={"ID":"9135ebe4-7ef2-4383-8a5a-3df3b11070eb","Type":"ContainerStarted","Data":"c6417cccffa1d237c27e45c9decd77648145e4ae8d4d047ff180a26b00310a49"}
Nov 01 07:04:40.201240 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:40.201205    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7" event={"ID":"9135ebe4-7ef2-4383-8a5a-3df3b11070eb","Type":"ContainerStarted","Data":"53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a"}
Nov 01 07:04:40.201764 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:40.201287    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:40.321352 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:40.321300    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7" podStartSLOduration=2.459878733 podStartE2EDuration="3.321285526s" totalImagesPullingTime="861.406793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:04:40.31955059 +0000 UTC m=+4717.197523998" watchObservedRunningTime="2026-11-01 07:04:40.321285526 +0000 UTC m=+4717.199258936"
Nov 01 07:04:45.316232 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:45.316199    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:04:46.212657 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:46.212626    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:04:47.186329 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:04:47.186301    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:04:47.544619 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:04:47.544570    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:05:01.047199 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:01.047162    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"]
Nov 01 07:05:01.096578 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:01.096546    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hclkd"]
Nov 01 07:05:01.234481 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:01.234453    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1805529-9dbf-4fe8-926e-395ce2e8fbfa" path="/var/lib/kubelet/pods/c1805529-9dbf-4fe8-926e-395ce2e8fbfa/volumes"
Nov 01 07:05:17.544139 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:05:17.544099    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:05:29.302858 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:29.302806    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"]
Nov 01 07:05:34.303383 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.303322    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r" podUID="128dea24-5bb0-4858-bfda-b74cde8ed32d" containerName="payload-pre-processing" containerID="cri-o://bdd2584d56f01193e1c8e72506c24312da78b5bc5ec995219bbbd89be123c42a" gracePeriod=25
Nov 01 07:05:34.502341 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.502297    5510 generic.go:356] "Generic (PLEG): container finished" podID="128dea24-5bb0-4858-bfda-b74cde8ed32d" containerID="bdd2584d56f01193e1c8e72506c24312da78b5bc5ec995219bbbd89be123c42a" exitCode=0
Nov 01 07:05:34.502517 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.502372    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r" event={"ID":"128dea24-5bb0-4858-bfda-b74cde8ed32d","Type":"ContainerDied","Data":"bdd2584d56f01193e1c8e72506c24312da78b5bc5ec995219bbbd89be123c42a"}
Nov 01 07:05:34.561524 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.561453    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:05:34.655777 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.655736    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128dea24-5bb0-4858-bfda-b74cde8ed32d-kube-api-access-sh8x4\" (UniqueName: \"kubernetes.io/projected/128dea24-5bb0-4858-bfda-b74cde8ed32d-kube-api-access-sh8x4\") pod \"128dea24-5bb0-4858-bfda-b74cde8ed32d\" (UID: \"128dea24-5bb0-4858-bfda-b74cde8ed32d\") "
Nov 01 07:05:34.655970 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.655789    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128dea24-5bb0-4858-bfda-b74cde8ed32d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dea24-5bb0-4858-bfda-b74cde8ed32d-plugins-config-volume\") pod \"128dea24-5bb0-4858-bfda-b74cde8ed32d\" (UID: \"128dea24-5bb0-4858-bfda-b74cde8ed32d\") "
Nov 01 07:05:34.656234 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.656208    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128dea24-5bb0-4858-bfda-b74cde8ed32d-plugins-config-volume" pod "128dea24-5bb0-4858-bfda-b74cde8ed32d" (UID: "128dea24-5bb0-4858-bfda-b74cde8ed32d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:05:34.658095 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.658057    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128dea24-5bb0-4858-bfda-b74cde8ed32d-kube-api-access-sh8x4" pod "128dea24-5bb0-4858-bfda-b74cde8ed32d" (UID: "128dea24-5bb0-4858-bfda-b74cde8ed32d"). InnerVolumeSpecName "kube-api-access-sh8x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:05:34.757395 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.757350    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh8x4\" (UniqueName: \"kubernetes.io/projected/128dea24-5bb0-4858-bfda-b74cde8ed32d-kube-api-access-sh8x4\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:34.757395 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.757385    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/128dea24-5bb0-4858-bfda-b74cde8ed32d-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:34.822382 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.822285    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"]
Nov 01 07:05:34.822566 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:34.822521    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7" podUID="9135ebe4-7ef2-4383-8a5a-3df3b11070eb" containerName="maas-api" containerID="cri-o://53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a" gracePeriod=30
Nov 01 07:05:35.085952 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.085928    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:05:35.159735 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.159703    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-metrics-certs\") pod \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") "
Nov 01 07:05:35.159950 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.159779    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls\") pod \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") "
Nov 01 07:05:35.159950 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.159809    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-openshift-service-ca\") pod \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") "
Nov 01 07:05:35.160106 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.159970    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-kube-api-access-shd7k\" (UniqueName: \"kubernetes.io/projected/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-kube-api-access-shd7k\") pod \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\" (UID: \"9135ebe4-7ef2-4383-8a5a-3df3b11070eb\") "
Nov 01 07:05:35.160298 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.160272    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-openshift-service-ca" pod "9135ebe4-7ef2-4383-8a5a-3df3b11070eb" (UID: "9135ebe4-7ef2-4383-8a5a-3df3b11070eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:05:35.162203 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.162173    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-metrics-certs" pod "9135ebe4-7ef2-4383-8a5a-3df3b11070eb" (UID: "9135ebe4-7ef2-4383-8a5a-3df3b11070eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:05:35.162285 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.162214    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-kube-api-access-shd7k" pod "9135ebe4-7ef2-4383-8a5a-3df3b11070eb" (UID: "9135ebe4-7ef2-4383-8a5a-3df3b11070eb"). InnerVolumeSpecName "kube-api-access-shd7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:05:35.162285 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.162229    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls" pod "9135ebe4-7ef2-4383-8a5a-3df3b11070eb" (UID: "9135ebe4-7ef2-4383-8a5a-3df3b11070eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:05:35.261067 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.261039    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shd7k\" (UniqueName: \"kubernetes.io/projected/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-kube-api-access-shd7k\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:35.261067 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.261067    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:35.261241 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.261080    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:35.261241 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.261092    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9135ebe4-7ef2-4383-8a5a-3df3b11070eb-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:35.508936 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.508912    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"
Nov 01 07:05:35.509460 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.508954    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r" event={"ID":"128dea24-5bb0-4858-bfda-b74cde8ed32d","Type":"ContainerDied","Data":"8a76881ccf630a9d67c2a01399f372490200bf4ea93231d87f61e89ae489f5f4"}
Nov 01 07:05:35.509460 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.509013    5510 scope.go:122] "RemoveContainer" containerID="bdd2584d56f01193e1c8e72506c24312da78b5bc5ec995219bbbd89be123c42a"
Nov 01 07:05:35.510419 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.510341    5510 generic.go:356] "Generic (PLEG): container finished" podID="9135ebe4-7ef2-4383-8a5a-3df3b11070eb" containerID="53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a" exitCode=0
Nov 01 07:05:35.510419 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.510400    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"
Nov 01 07:05:35.510525 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.510441    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7" event={"ID":"9135ebe4-7ef2-4383-8a5a-3df3b11070eb","Type":"ContainerDied","Data":"53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a"}
Nov 01 07:05:35.510525 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.510482    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7" event={"ID":"9135ebe4-7ef2-4383-8a5a-3df3b11070eb","Type":"ContainerDied","Data":"c6417cccffa1d237c27e45c9decd77648145e4ae8d4d047ff180a26b00310a49"}
Nov 01 07:05:35.518918 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.518900    5510 scope.go:122] "RemoveContainer" containerID="53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a"
Nov 01 07:05:35.527448 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.527428    5510 scope.go:122] "RemoveContainer" containerID="53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a"
Nov 01 07:05:35.527749 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:05:35.527726    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a\": container with ID starting with 53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a not found: ID does not exist" containerID="53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a"
Nov 01 07:05:35.527797 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.527760    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a"} err="failed to get container status \"53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a\": rpc error: code = NotFound desc = could not find container \"53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a\": container with ID starting with 53f4f3ec0d0d5476815b2401027a1a80401fdcdedd4063632c647df930ec280a not found: ID does not exist"
Nov 01 07:05:35.672515 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.672431    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"]
Nov 01 07:05:35.675839 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.675801    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9af193c3-59667c9b79-jjr8r"]
Nov 01 07:05:35.790870 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.790270    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"]
Nov 01 07:05:35.800926 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:35.800897    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9af193c3-848bbb48-vdwx7"]
Nov 01 07:05:37.235333 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:37.235291    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128dea24-5bb0-4858-bfda-b74cde8ed32d" path="/var/lib/kubelet/pods/128dea24-5bb0-4858-bfda-b74cde8ed32d/volumes"
Nov 01 07:05:37.235761 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:37.235746    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9135ebe4-7ef2-4383-8a5a-3df3b11070eb" path="/var/lib/kubelet/pods/9135ebe4-7ef2-4383-8a5a-3df3b11070eb/volumes"
Nov 01 07:05:39.413223 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:39.413189    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:05:47.725907 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:05:47.725190    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:05:50.002591 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:50.002557    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:05:51.259003 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.258971    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"]
Nov 01 07:05:51.259424 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.259405    5510 cpu_manager.go:414] "containerMap: removing container" podUID="128dea24-5bb0-4858-bfda-b74cde8ed32d" containerName="payload-pre-processing"
Nov 01 07:05:51.259487 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.259429    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128dea24-5bb0-4858-bfda-b74cde8ed32d" containerName="payload-pre-processing"
Nov 01 07:05:51.259487 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.259456    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9135ebe4-7ef2-4383-8a5a-3df3b11070eb" containerName="maas-api"
Nov 01 07:05:51.259487 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.259465    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9135ebe4-7ef2-4383-8a5a-3df3b11070eb" containerName="maas-api"
Nov 01 07:05:51.259594 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.259545    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="128dea24-5bb0-4858-bfda-b74cde8ed32d" containerName="payload-pre-processing"
Nov 01 07:05:51.259594 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.259554    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9135ebe4-7ef2-4383-8a5a-3df3b11070eb" containerName="maas-api"
Nov 01 07:05:51.262762 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.262746    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.315384 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.315360    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:05:51.315384 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.315360    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:05:51.347035 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.347000    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqlm6\" (UniqueName: \"kubernetes.io/projected/8fa8486c-e622-4b2a-bb1c-24b030da4531-kube-api-access-hqlm6\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.347198 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.347055    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.347198 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.347078    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.364776 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.364745    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:05:51.364935 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.364904    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:05:51.367031 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.367012    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"]
Nov 01 07:05:51.447930 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.447891    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqlm6\" (UniqueName: \"kubernetes.io/projected/8fa8486c-e622-4b2a-bb1c-24b030da4531-kube-api-access-hqlm6\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.448087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.447960    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.448087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.447991    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.448664 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.448638    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.450406 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.450389    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.508879 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.508854    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqlm6\" (UniqueName: \"kubernetes.io/projected/8fa8486c-e622-4b2a-bb1c-24b030da4531-kube-api-access-hqlm6\") pod \"maas-api-revoke-sub-f904bff9-rr77f\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.574806 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.574732    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:51.946811 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:51.946778    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"]
Nov 01 07:05:51.948138 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:05:51.948096    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa8486c_e622_4b2a_bb1c_24b030da4531.slice/crio-159255be342fba5eabe22d160ccf10debe13a620db2ec12854292a0b18d02e89 WatchSource:0}: Error finding container 159255be342fba5eabe22d160ccf10debe13a620db2ec12854292a0b18d02e89: Status 404 returned error can't find the container with id 159255be342fba5eabe22d160ccf10debe13a620db2ec12854292a0b18d02e89
Nov 01 07:05:52.629766 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:52.629677    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f" event={"ID":"8fa8486c-e622-4b2a-bb1c-24b030da4531","Type":"ContainerStarted","Data":"6afbdcedf87e146250f3a6ff98900c7689c8720f86380dd6f59123e3b823fd3d"}
Nov 01 07:05:52.629766 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:52.629732    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f" event={"ID":"8fa8486c-e622-4b2a-bb1c-24b030da4531","Type":"ContainerStarted","Data":"159255be342fba5eabe22d160ccf10debe13a620db2ec12854292a0b18d02e89"}
Nov 01 07:05:52.747735 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:52.747681    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f" podStartSLOduration=1.7476676439999999 podStartE2EDuration="1.747667644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:52.745376602 +0000 UTC m=+4789.623350014" watchObservedRunningTime="2026-11-01 07:05:52.747667644 +0000 UTC m=+4789.625641055"
Nov 01 07:05:53.635988 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:53.635952    5510 generic.go:356] "Generic (PLEG): container finished" podID="8fa8486c-e622-4b2a-bb1c-24b030da4531" containerID="6afbdcedf87e146250f3a6ff98900c7689c8720f86380dd6f59123e3b823fd3d" exitCode=0
Nov 01 07:05:53.636433 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:53.636047    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f" event={"ID":"8fa8486c-e622-4b2a-bb1c-24b030da4531","Type":"ContainerDied","Data":"6afbdcedf87e146250f3a6ff98900c7689c8720f86380dd6f59123e3b823fd3d"}
Nov 01 07:05:54.783201 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.783164    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:54.877802 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.877775    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-openshift-ca\") pod \"8fa8486c-e622-4b2a-bb1c-24b030da4531\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") "
Nov 01 07:05:54.877996 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.877823    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-xks-ca\") pod \"8fa8486c-e622-4b2a-bb1c-24b030da4531\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") "
Nov 01 07:05:54.877996 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.877931    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa8486c-e622-4b2a-bb1c-24b030da4531-kube-api-access-hqlm6\" (UniqueName: \"kubernetes.io/projected/8fa8486c-e622-4b2a-bb1c-24b030da4531-kube-api-access-hqlm6\") pod \"8fa8486c-e622-4b2a-bb1c-24b030da4531\" (UID: \"8fa8486c-e622-4b2a-bb1c-24b030da4531\") "
Nov 01 07:05:54.878178 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.878157    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-openshift-ca" pod "8fa8486c-e622-4b2a-bb1c-24b030da4531" (UID: "8fa8486c-e622-4b2a-bb1c-24b030da4531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:05:54.878362 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.878346    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:54.879813 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.879788    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-xks-ca" pod "8fa8486c-e622-4b2a-bb1c-24b030da4531" (UID: "8fa8486c-e622-4b2a-bb1c-24b030da4531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:05:54.880042 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.880023    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa8486c-e622-4b2a-bb1c-24b030da4531-kube-api-access-hqlm6" pod "8fa8486c-e622-4b2a-bb1c-24b030da4531" (UID: "8fa8486c-e622-4b2a-bb1c-24b030da4531"). InnerVolumeSpecName "kube-api-access-hqlm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:05:54.978887 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.978855    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa8486c-e622-4b2a-bb1c-24b030da4531-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:54.978887 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:54.978877    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqlm6\" (UniqueName: \"kubernetes.io/projected/8fa8486c-e622-4b2a-bb1c-24b030da4531-kube-api-access-hqlm6\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:55.649963 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:55.649879    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f" event={"ID":"8fa8486c-e622-4b2a-bb1c-24b030da4531","Type":"ContainerDied","Data":"159255be342fba5eabe22d160ccf10debe13a620db2ec12854292a0b18d02e89"}
Nov 01 07:05:55.649963 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:55.649946    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159255be342fba5eabe22d160ccf10debe13a620db2ec12854292a0b18d02e89"
Nov 01 07:05:55.649963 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:55.649894    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"
Nov 01 07:05:59.046348 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:59.046317    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"]
Nov 01 07:05:59.095690 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:59.095660    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc65cc-jf8vn"]
Nov 01 07:05:59.234585 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:05:59.234545    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06" path="/var/lib/kubelet/pods/6a324ef6-d72c-4c7b-9f8a-f5339e7f0e06/volumes"
Nov 01 07:06:04.190135 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:04.190096    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:06:04.889667 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:04.889635    5510 scope.go:122] "RemoveContainer" containerID="0fd159795c894e2d06761a68e081c631c19227108b6e8f527afcf8ab1e448246"
Nov 01 07:06:09.055740 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:09.055696    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"]
Nov 01 07:06:09.104749 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:09.104708    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f7e4b7-cgdrl"]
Nov 01 07:06:09.234086 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:09.234055    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3828f0-cef8-4c63-97fb-ba04539606b0" path="/var/lib/kubelet/pods/cd3828f0-cef8-4c63-97fb-ba04539606b0/volumes"
Nov 01 07:06:13.869890 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:13.869848    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:06:17.543782 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:06:17.543743    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:06:20.049613 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:20.049565    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"]
Nov 01 07:06:20.099852 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:20.099581    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec2b1d6-5jcks"]
Nov 01 07:06:21.265855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:21.249224    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb67b6a3-bc08-4eab-bfda-d50821eeb947" path="/var/lib/kubelet/pods/fb67b6a3-bc08-4eab-bfda-d50821eeb947/volumes"
Nov 01 07:06:23.194519 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:23.194481    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:06:25.019590 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.019553    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"]
Nov 01 07:06:25.020013 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.020002    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8486c-e622-4b2a-bb1c-24b030da4531" containerName="revoke-keys"
Nov 01 07:06:25.020277 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.020014    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8486c-e622-4b2a-bb1c-24b030da4531" containerName="revoke-keys"
Nov 01 07:06:25.020277 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.020073    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa8486c-e622-4b2a-bb1c-24b030da4531" containerName="revoke-keys"
Nov 01 07:06:25.023344 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.023326    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.036900 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.036878    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.037002 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.036906    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfh7\" (UniqueName: \"kubernetes.io/projected/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-kube-api-access-jtfh7\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.037002 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.036943    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.075241 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.075214    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:06:25.075241 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.075215    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:06:25.124857 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.124798    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:06:25.124857 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.124798    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:06:25.127119 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.127094    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"]
Nov 01 07:06:25.138363 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.138339    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.138501 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.138372    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfh7\" (UniqueName: \"kubernetes.io/projected/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-kube-api-access-jtfh7\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.138501 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.138453    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.139168 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.139100    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.140878 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.140859    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.199082 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.199050    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfh7\" (UniqueName: \"kubernetes.io/projected/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-kube-api-access-jtfh7\") pod \"maas-api-revoke-sub-f03629db-r7vjt\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.335870 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.335764    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:25.709704 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.709680    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"]
Nov 01 07:06:25.712092 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:06:25.712065    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9339f07a_fcde_4fd9_b3fc_1fb8e619eb20.slice/crio-ffa3f9f4f27af06c56e828e68fd1eee19f04216faf4b316330b3fe0e0be3f30f WatchSource:0}: Error finding container ffa3f9f4f27af06c56e828e68fd1eee19f04216faf4b316330b3fe0e0be3f30f: Status 404 returned error can't find the container with id ffa3f9f4f27af06c56e828e68fd1eee19f04216faf4b316330b3fe0e0be3f30f
Nov 01 07:06:25.875120 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:25.875090    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt" event={"ID":"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20","Type":"ContainerStarted","Data":"ffa3f9f4f27af06c56e828e68fd1eee19f04216faf4b316330b3fe0e0be3f30f"}
Nov 01 07:06:26.881869 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:26.881807    5510 generic.go:356] "Generic (PLEG): container finished" podID="9339f07a-fcde-4fd9-b3fc-1fb8e619eb20" containerID="f252aa7affee3201c9f787b92b925d9027b1f9c789708f577b45932c877e0724" exitCode=0
Nov 01 07:06:26.882326 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:26.881858    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt" event={"ID":"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20","Type":"ContainerDied","Data":"f252aa7affee3201c9f787b92b925d9027b1f9c789708f577b45932c877e0724"}
Nov 01 07:06:28.020694 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.020667    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:28.066282 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.066249    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-kube-api-access-jtfh7\" (UniqueName: \"kubernetes.io/projected/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-kube-api-access-jtfh7\") pod \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") "
Nov 01 07:06:28.066451 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.066299    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-openshift-ca\") pod \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") "
Nov 01 07:06:28.066451 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.066323    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-xks-ca\") pod \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\" (UID: \"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20\") "
Nov 01 07:06:28.066731 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.066708    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-openshift-ca" pod "9339f07a-fcde-4fd9-b3fc-1fb8e619eb20" (UID: "9339f07a-fcde-4fd9-b3fc-1fb8e619eb20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:28.068568 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.068540    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-xks-ca" pod "9339f07a-fcde-4fd9-b3fc-1fb8e619eb20" (UID: "9339f07a-fcde-4fd9-b3fc-1fb8e619eb20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:28.068706 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.068616    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-kube-api-access-jtfh7" pod "9339f07a-fcde-4fd9-b3fc-1fb8e619eb20" (UID: "9339f07a-fcde-4fd9-b3fc-1fb8e619eb20"). InnerVolumeSpecName "kube-api-access-jtfh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:28.167603 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.167503    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtfh7\" (UniqueName: \"kubernetes.io/projected/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-kube-api-access-jtfh7\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:28.167603 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.167536    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:28.167603 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.167553    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:28.893979 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.893949    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"
Nov 01 07:06:28.894166 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.893986    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt" event={"ID":"9339f07a-fcde-4fd9-b3fc-1fb8e619eb20","Type":"ContainerDied","Data":"ffa3f9f4f27af06c56e828e68fd1eee19f04216faf4b316330b3fe0e0be3f30f"}
Nov 01 07:06:28.894166 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:28.894054    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa3f9f4f27af06c56e828e68fd1eee19f04216faf4b316330b3fe0e0be3f30f"
Nov 01 07:06:31.046608 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:31.046578    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"]
Nov 01 07:06:31.096168 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:31.096133    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b2d386-cfqmm"]
Nov 01 07:06:31.234356 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:31.234322    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7eb1ea3-87e3-4c7c-a786-d4efa8838659" path="/var/lib/kubelet/pods/e7eb1ea3-87e3-4c7c-a786-d4efa8838659/volumes"
Nov 01 07:06:31.680624 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:31.680596    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:06:42.049298 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:42.049256    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"]
Nov 01 07:06:42.096522 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:42.096486    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff1785a-qrzvp"]
Nov 01 07:06:43.234556 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:43.234524    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf5b3ab-7f65-4106-93e8-fc556868feb9" path="/var/lib/kubelet/pods/2bf5b3ab-7f65-4106-93e8-fc556868feb9/volumes"
Nov 01 07:06:47.543967 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:06:47.543932    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:06:53.704299 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.704262    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"]
Nov 01 07:06:53.704771 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.704666    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9339f07a-fcde-4fd9-b3fc-1fb8e619eb20" containerName="revoke-keys"
Nov 01 07:06:53.704771 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.704677    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9339f07a-fcde-4fd9-b3fc-1fb8e619eb20" containerName="revoke-keys"
Nov 01 07:06:53.704771 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.704753    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9339f07a-fcde-4fd9-b3fc-1fb8e619eb20" containerName="revoke-keys"
Nov 01 07:06:53.707963 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.707946    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:53.760014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.759992    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:06:53.809408 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.809368    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:06:53.809408 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.809383    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:06:53.809408 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.809405    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:06:53.811365 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.811347    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"]
Nov 01 07:06:53.864245 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.864220    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjl9\" (UniqueName: \"kubernetes.io/projected/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-kube-api-access-fgjl9\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:53.864381 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.864267    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:53.864381 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.864347    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:53.965636 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.965607    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjl9\" (UniqueName: \"kubernetes.io/projected/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-kube-api-access-fgjl9\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:53.965781 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.965669    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:53.965781 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.965723    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:53.966415 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.966396    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:53.968120 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:53.968097    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:54.026592 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:54.026566    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjl9\" (UniqueName: \"kubernetes.io/projected/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-kube-api-access-fgjl9\") pod \"maas-api-revoke-sub-0165d7d2-w5gwz\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:54.048745 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:54.048716    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"]
Nov 01 07:06:54.051133 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:54.051111    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc2b22e-wfrjm"]
Nov 01 07:06:54.320601 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:54.320525    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:54.701242 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:54.701101    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"]
Nov 01 07:06:54.703479 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:06:54.703447    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea8d3ba_c60a_478e_bab4_48ac542c3df7.slice/crio-7140e13ae0a4576b6b695bbc7c8c4de9335c63a6e55525c39a8dc72959ea9592 WatchSource:0}: Error finding container 7140e13ae0a4576b6b695bbc7c8c4de9335c63a6e55525c39a8dc72959ea9592: Status 404 returned error can't find the container with id 7140e13ae0a4576b6b695bbc7c8c4de9335c63a6e55525c39a8dc72959ea9592
Nov 01 07:06:55.066795 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:55.066763    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz" event={"ID":"1ea8d3ba-c60a-478e-bab4-48ac542c3df7","Type":"ContainerStarted","Data":"f92acedca10a75c28a5c30227a5bdab272142e2170bb0f33b137e22a3db596b2"}
Nov 01 07:06:55.067159 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:55.066819    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz" event={"ID":"1ea8d3ba-c60a-478e-bab4-48ac542c3df7","Type":"ContainerStarted","Data":"7140e13ae0a4576b6b695bbc7c8c4de9335c63a6e55525c39a8dc72959ea9592"}
Nov 01 07:06:55.182990 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:55.182930    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz" podStartSLOduration=2.182917256 podStartE2EDuration="2.182917256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:55.181252344 +0000 UTC m=+4852.059225767" watchObservedRunningTime="2026-11-01 07:06:55.182917256 +0000 UTC m=+4852.060890665"
Nov 01 07:06:55.241487 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:55.241406    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4261dfb-4974-436f-9726-c5516fc9a02c" path="/var/lib/kubelet/pods/b4261dfb-4974-436f-9726-c5516fc9a02c/volumes"
Nov 01 07:06:56.074112 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:56.074077    5510 generic.go:356] "Generic (PLEG): container finished" podID="1ea8d3ba-c60a-478e-bab4-48ac542c3df7" containerID="f92acedca10a75c28a5c30227a5bdab272142e2170bb0f33b137e22a3db596b2" exitCode=0
Nov 01 07:06:56.074514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:56.074157    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz" event={"ID":"1ea8d3ba-c60a-478e-bab4-48ac542c3df7","Type":"ContainerDied","Data":"f92acedca10a75c28a5c30227a5bdab272142e2170bb0f33b137e22a3db596b2"}
Nov 01 07:06:57.225410 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.225386    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:06:57.399067 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.398992    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-xks-ca\") pod \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") "
Nov 01 07:06:57.399067 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.399062    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-openshift-ca\") pod \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") "
Nov 01 07:06:57.399291 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.399117    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-kube-api-access-fgjl9\" (UniqueName: \"kubernetes.io/projected/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-kube-api-access-fgjl9\") pod \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\" (UID: \"1ea8d3ba-c60a-478e-bab4-48ac542c3df7\") "
Nov 01 07:06:57.399449 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.399422    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-openshift-ca" pod "1ea8d3ba-c60a-478e-bab4-48ac542c3df7" (UID: "1ea8d3ba-c60a-478e-bab4-48ac542c3df7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:57.401223 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.401198    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-xks-ca" pod "1ea8d3ba-c60a-478e-bab4-48ac542c3df7" (UID: "1ea8d3ba-c60a-478e-bab4-48ac542c3df7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:57.401387 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.401366    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-kube-api-access-fgjl9" pod "1ea8d3ba-c60a-478e-bab4-48ac542c3df7" (UID: "1ea8d3ba-c60a-478e-bab4-48ac542c3df7"). InnerVolumeSpecName "kube-api-access-fgjl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:57.499850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.499772    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:57.499850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.499818    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgjl9\" (UniqueName: \"kubernetes.io/projected/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-kube-api-access-fgjl9\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:57.499850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:57.499853    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea8d3ba-c60a-478e-bab4-48ac542c3df7-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:58.086438 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:58.086404    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz" event={"ID":"1ea8d3ba-c60a-478e-bab4-48ac542c3df7","Type":"ContainerDied","Data":"7140e13ae0a4576b6b695bbc7c8c4de9335c63a6e55525c39a8dc72959ea9592"}
Nov 01 07:06:58.086615 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:58.086456    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7140e13ae0a4576b6b695bbc7c8c4de9335c63a6e55525c39a8dc72959ea9592"
Nov 01 07:06:58.086615 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:06:58.086416    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"
Nov 01 07:07:04.047091 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:04.047056    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"]
Nov 01 07:07:04.096585 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:04.096558    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25cc81f-z5p7h"]
Nov 01 07:07:04.918529 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:04.918500    5510 scope.go:122] "RemoveContainer" containerID="241120fe1e402fbdcd4e6549d8e958f31d7dccec45719d300ea293e79a8b8a07"
Nov 01 07:07:05.234684 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.234655    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f605c2-6b4d-454f-9a08-93c15644877c" path="/var/lib/kubelet/pods/36f605c2-6b4d-454f-9a08-93c15644877c/volumes"
Nov 01 07:07:05.546741 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.546656    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"]
Nov 01 07:07:05.547141 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.547124    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1ea8d3ba-c60a-478e-bab4-48ac542c3df7" containerName="revoke-keys"
Nov 01 07:07:05.547201 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.547142    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea8d3ba-c60a-478e-bab4-48ac542c3df7" containerName="revoke-keys"
Nov 01 07:07:05.547244 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.547236    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea8d3ba-c60a-478e-bab4-48ac542c3df7" containerName="revoke-keys"
Nov 01 07:07:05.551894 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.551878    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.557694 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.557670    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.557846 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.557746    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.557846 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.557789    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6xq\" (UniqueName: \"kubernetes.io/projected/13525afa-0790-49a6-94f9-f13ba73074f3-kube-api-access-4n6xq\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.605979 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.605938    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:07:05.655235 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.655205    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:07:05.655235 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.655207    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:07:05.655498 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.655331    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:07:05.657222 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.657180    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"]
Nov 01 07:07:05.658891 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.658869    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.658959 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.658922    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6xq\" (UniqueName: \"kubernetes.io/projected/13525afa-0790-49a6-94f9-f13ba73074f3-kube-api-access-4n6xq\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.659058 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.659042    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.659487 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.659469    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.661558 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.661532    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.719432 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.719407    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6xq\" (UniqueName: \"kubernetes.io/projected/13525afa-0790-49a6-94f9-f13ba73074f3-kube-api-access-4n6xq\") pod \"maas-api-revoke-sub-6b19338f-xlppn\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:05.865193 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:05.865114    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:06.245058 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:06.245033    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"]
Nov 01 07:07:06.247325 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:07:06.247298    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13525afa_0790_49a6_94f9_f13ba73074f3.slice/crio-18e5bb9cf85765e36689163f270c0e06e98e5a8254108ce097e22853af8396b6 WatchSource:0}: Error finding container 18e5bb9cf85765e36689163f270c0e06e98e5a8254108ce097e22853af8396b6: Status 404 returned error can't find the container with id 18e5bb9cf85765e36689163f270c0e06e98e5a8254108ce097e22853af8396b6
Nov 01 07:07:07.135558 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:07.135520    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn" event={"ID":"13525afa-0790-49a6-94f9-f13ba73074f3","Type":"ContainerStarted","Data":"309cea23fa4863867e2974018e40143d318994b23cad41cb0a50894710393d87"}
Nov 01 07:07:07.135736 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:07.135610    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn" event={"ID":"13525afa-0790-49a6-94f9-f13ba73074f3","Type":"ContainerStarted","Data":"18e5bb9cf85765e36689163f270c0e06e98e5a8254108ce097e22853af8396b6"}
Nov 01 07:07:07.251923 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:07.251878    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn" podStartSLOduration=2.251864638 podStartE2EDuration="2.251864638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:07.250293475 +0000 UTC m=+4864.128266888" watchObservedRunningTime="2026-11-01 07:07:07.251864638 +0000 UTC m=+4864.129838027"
Nov 01 07:07:08.142093 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:08.142060    5510 generic.go:356] "Generic (PLEG): container finished" podID="13525afa-0790-49a6-94f9-f13ba73074f3" containerID="309cea23fa4863867e2974018e40143d318994b23cad41cb0a50894710393d87" exitCode=0
Nov 01 07:07:08.142093 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:08.142095    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn" event={"ID":"13525afa-0790-49a6-94f9-f13ba73074f3","Type":"ContainerDied","Data":"309cea23fa4863867e2974018e40143d318994b23cad41cb0a50894710393d87"}
Nov 01 07:07:09.282496 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.282471    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:09.302685 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.302660    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-xks-ca\") pod \"13525afa-0790-49a6-94f9-f13ba73074f3\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") "
Nov 01 07:07:09.302815 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.302761    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13525afa-0790-49a6-94f9-f13ba73074f3-kube-api-access-4n6xq\" (UniqueName: \"kubernetes.io/projected/13525afa-0790-49a6-94f9-f13ba73074f3-kube-api-access-4n6xq\") pod \"13525afa-0790-49a6-94f9-f13ba73074f3\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") "
Nov 01 07:07:09.302894 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.302821    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-openshift-ca\") pod \"13525afa-0790-49a6-94f9-f13ba73074f3\" (UID: \"13525afa-0790-49a6-94f9-f13ba73074f3\") "
Nov 01 07:07:09.303124 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.303100    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-openshift-ca" pod "13525afa-0790-49a6-94f9-f13ba73074f3" (UID: "13525afa-0790-49a6-94f9-f13ba73074f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:09.303214 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.303199    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:09.305037 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.305015    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-xks-ca" pod "13525afa-0790-49a6-94f9-f13ba73074f3" (UID: "13525afa-0790-49a6-94f9-f13ba73074f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:07:09.305037 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.305018    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13525afa-0790-49a6-94f9-f13ba73074f3-kube-api-access-4n6xq" pod "13525afa-0790-49a6-94f9-f13ba73074f3" (UID: "13525afa-0790-49a6-94f9-f13ba73074f3"). InnerVolumeSpecName "kube-api-access-4n6xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:09.404242 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.404172    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n6xq\" (UniqueName: \"kubernetes.io/projected/13525afa-0790-49a6-94f9-f13ba73074f3-kube-api-access-4n6xq\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:09.404242 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:09.404199    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13525afa-0790-49a6-94f9-f13ba73074f3-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:10.153401 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:10.153373    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"
Nov 01 07:07:10.153595 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:10.153405    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn" event={"ID":"13525afa-0790-49a6-94f9-f13ba73074f3","Type":"ContainerDied","Data":"18e5bb9cf85765e36689163f270c0e06e98e5a8254108ce097e22853af8396b6"}
Nov 01 07:07:10.153595 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:10.153458    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e5bb9cf85765e36689163f270c0e06e98e5a8254108ce097e22853af8396b6"
Nov 01 07:07:15.048883 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:15.048754    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"]
Nov 01 07:07:15.098143 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:15.098110    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c727f1-2gh6r"]
Nov 01 07:07:15.233400 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:15.233370    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b0fc50-0c6f-40bb-bb4b-d54eff851a9a" path="/var/lib/kubelet/pods/68b0fc50-0c6f-40bb-bb4b-d54eff851a9a/volumes"
Nov 01 07:07:17.543489 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:07:17.543454    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:07:18.384062 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:18.384028    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:07:21.475693 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.475658    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"]
Nov 01 07:07:21.476104 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.476076    5510 cpu_manager.go:414] "containerMap: removing container" podUID="13525afa-0790-49a6-94f9-f13ba73074f3" containerName="revoke-keys"
Nov 01 07:07:21.476104 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.476086    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13525afa-0790-49a6-94f9-f13ba73074f3" containerName="revoke-keys"
Nov 01 07:07:21.476178 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.476160    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="13525afa-0790-49a6-94f9-f13ba73074f3" containerName="revoke-keys"
Nov 01 07:07:21.479222 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.479206    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:21.548648 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.548614    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j899v\" (UniqueName: \"kubernetes.io/projected/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-kube-api-access-j899v\") pod \"e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw\" (UID: \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\") " pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:21.548803 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.548730    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-tls-certs\") pod \"e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw\" (UID: \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\") " pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:21.554087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.554048    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-fade3a71-epp-sa-dockercfg-r6r5g\""
Nov 01 07:07:21.554087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.554050    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-fade3a71-kserve-self-signed-certs\""
Nov 01 07:07:21.582599 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.582571    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"]
Nov 01 07:07:21.650079 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.650047    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j899v\" (UniqueName: \"kubernetes.io/projected/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-kube-api-access-j899v\") pod \"e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw\" (UID: \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\") " pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:21.650259 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.650123    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-tls-certs\") pod \"e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw\" (UID: \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\") " pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:21.652811 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.652788    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-tls-certs\") pod \"e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw\" (UID: \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\") " pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:21.710894 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.710862    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j899v\" (UniqueName: \"kubernetes.io/projected/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-kube-api-access-j899v\") pod \"e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw\" (UID: \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\") " pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:21.791044 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:21.790967    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:22.162054 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:22.162029    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"]
Nov 01 07:07:22.164392 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:07:22.164363    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44f7ac2_2ebd_48c3_a509_c3b332c00c6a.slice/crio-f2736f2f22ecfefd3c79d1735997c8a291337f6f1bc89b649fd8f7215586fd09 WatchSource:0}: Error finding container f2736f2f22ecfefd3c79d1735997c8a291337f6f1bc89b649fd8f7215586fd09: Status 404 returned error can't find the container with id f2736f2f22ecfefd3c79d1735997c8a291337f6f1bc89b649fd8f7215586fd09
Nov 01 07:07:22.237497 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:22.237460    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" event={"ID":"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a","Type":"ContainerStarted","Data":"f2736f2f22ecfefd3c79d1735997c8a291337f6f1bc89b649fd8f7215586fd09"}
Nov 01 07:07:24.251050 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:24.250973    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" event={"ID":"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a","Type":"ContainerStarted","Data":"acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc"}
Nov 01 07:07:24.251430 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:24.251053    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:24.389093 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:24.389035    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" podStartSLOduration=1.575550124 podStartE2EDuration="3.38901766s" totalImagesPullingTime="1.813467536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:24.388227271 +0000 UTC m=+4881.266200707" watchObservedRunningTime="2026-11-01 07:07:24.38901766 +0000 UTC m=+4881.266991072"
Nov 01 07:07:31.998625 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:31.998589    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:07:36.093767 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:36.093733    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"]
Nov 01 07:07:36.095817 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:36.095789    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f103976-rqlwq"]
Nov 01 07:07:37.234991 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:37.234959    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb000b1-b6a3-48bb-9644-562463e24f2c" path="/var/lib/kubelet/pods/1bb000b1-b6a3-48bb-9644-562463e24f2c/volumes"
Nov 01 07:07:47.047301 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:47.047272    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"]
Nov 01 07:07:47.096895 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:47.096864    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a445fb5-nknh8"]
Nov 01 07:07:47.234378 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:47.234342    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f" path="/var/lib/kubelet/pods/21b719ba-bfde-42a8-9e2d-0a15ef0f5e9f/volumes"
Nov 01 07:07:47.545393 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:07:47.545358    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:07:55.257672 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:55.257643    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:07:57.046376 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:57.046342    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"]
Nov 01 07:07:57.095428 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:57.095396    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dafed0a-v2qvs"]
Nov 01 07:07:57.235291 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:07:57.235258    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df" path="/var/lib/kubelet/pods/23a6f47e-8aeb-4b2d-8cc4-4cd23c0bb7df/volumes"
Nov 01 07:08:04.181818 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:04.181787    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:08:04.945295 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:04.945260    5510 scope.go:122] "RemoveContainer" containerID="b0d686d861e7861a3db81fbb0fd3793c1ca91088abd33ba5d721b88be6a924d5"
Nov 01 07:08:04.959463 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:04.959435    5510 scope.go:122] "RemoveContainer" containerID="f86795913b1831d428fe717fd9ebbe12cea995b25a6de29e95c2250152d61b70"
Nov 01 07:08:04.972099 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:04.972068    5510 scope.go:122] "RemoveContainer" containerID="e53be8a835ba0694a8caf589e4447bff06e235f6c233f2e28cf9faaf8c727745"
Nov 01 07:08:04.984202 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:04.984178    5510 scope.go:122] "RemoveContainer" containerID="6db88592b2d840da9cfd0fd301297b79035526050de029072c080c7d7587fe93"
Nov 01 07:08:04.997541 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:04.997517    5510 scope.go:122] "RemoveContainer" containerID="5ff3fffaf927fd72240a113db7931203abc199d9de51b8911c571f8e66f8f088"
Nov 01 07:08:05.013737 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:05.013713    5510 scope.go:122] "RemoveContainer" containerID="5a90a642611f6bc61113df2e33693745089b6b09d6bea85d81c2aae97a50392e"
Nov 01 07:08:06.637853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:06.635961    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-f9ttw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:08:06.637853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:06.636071    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-f9ttw" podUID="47d488d9-60bc-480d-b851-cea9d9a4edb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:08:08.046258 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:08.046225    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"]
Nov 01 07:08:08.095416 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:08.095388    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384805a7-hpzfd"]
Nov 01 07:08:09.235099 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:09.235064    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403604c1-aefb-48e3-bdd5-a1a4658bc1f2" path="/var/lib/kubelet/pods/403604c1-aefb-48e3-bdd5-a1a4658bc1f2/volumes"
Nov 01 07:08:17.543952 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:08:17.543914    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:08:23.046984 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:23.046947    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"]
Nov 01 07:08:23.096633 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:23.096591    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300feb70-k7nrn"]
Nov 01 07:08:23.234790 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:23.234759    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485f113d-f8a9-4c10-bb5d-520f0a6b02cc" path="/var/lib/kubelet/pods/485f113d-f8a9-4c10-bb5d-520f0a6b02cc/volumes"
Nov 01 07:08:28.163498 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:28.163458    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:08:28.334259 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:28.334229    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"]
Nov 01 07:08:34.048854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:34.047990    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"]
Nov 01 07:08:34.101841 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:34.101791    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d9d8e-p5btm"]
Nov 01 07:08:34.457950 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:34.457917    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:08:34.458245 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:34.458211    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:08:34.559049 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:34.559007    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:08:34.559369 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:34.559335    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:08:34.560235 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:34.560209    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:08:34.560411 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:34.560389    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:08:35.237864 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:35.237786    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abdda2b1-98f4-4ef3-a296-116a87e0e43d" path="/var/lib/kubelet/pods/abdda2b1-98f4-4ef3-a296-116a87e0e43d/volumes"
Nov 01 07:08:35.259596 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:35.259554    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:08:36.183777 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:36.183740    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:08:43.355111 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:43.355052    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerName="main" containerID="cri-o://acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc" gracePeriod=45
Nov 01 07:08:45.257817 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:45.257778    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:08:47.568269 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:08:47.568223    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:08:55.258304 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:55.258267    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:08:55.258675 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:08:55.258379    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:09:05.085048 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:05.085013    5510 scope.go:122] "RemoveContainer" containerID="ee41f2ef93e9222edea6f0d6bff23bd8e639c2c0bc7804127966f830cdbe2e7d"
Nov 01 07:09:05.094027 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:05.094004    5510 scope.go:122] "RemoveContainer" containerID="adf9e9fa26018a173d12fc1a30561b9c77930e63ec4ae0b672f72ced3d706cf7"
Nov 01 07:09:05.101764 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:05.101747    5510 scope.go:122] "RemoveContainer" containerID="a2071604407d154b36fc4f935d93af69dbc6dd44e7d27dcd3c6ee5d72f83cbf4"
Nov 01 07:09:05.109253 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:05.109226    5510 scope.go:122] "RemoveContainer" containerID="f08f3fcb7e5806eba91fef1bff8668f6a19d748cec8d5cdaa3b3c281942136c7"
Nov 01 07:09:05.257335 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:05.257302    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:09:13.598898 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.598876    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:09:13.786473 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.786433    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-kube-api-access-j899v\" (UniqueName: \"kubernetes.io/projected/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-kube-api-access-j899v\") pod \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\" (UID: \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\") "
Nov 01 07:09:13.786473 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.786479    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-tls-certs\") pod \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\" (UID: \"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a\") "
Nov 01 07:09:13.789079 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.789032    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-tls-certs" pod "f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" (UID: "f44f7ac2-2ebd-48c3-a509-c3b332c00c6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:09:13.789202 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.789081    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-kube-api-access-j899v" pod "f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" (UID: "f44f7ac2-2ebd-48c3-a509-c3b332c00c6a"). InnerVolumeSpecName "kube-api-access-j899v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:13.887195 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.887168    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j899v\" (UniqueName: \"kubernetes.io/projected/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-kube-api-access-j899v\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:13.887195 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.887193    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a-tls-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:13.932402 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.932374    5510 generic.go:356] "Generic (PLEG): container finished" podID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerID="acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc" exitCode=0
Nov 01 07:09:13.932505 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.932430    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"
Nov 01 07:09:13.932505 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.932430    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" event={"ID":"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a","Type":"ContainerDied","Data":"acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc"}
Nov 01 07:09:13.932505 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.932503    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw" event={"ID":"f44f7ac2-2ebd-48c3-a509-c3b332c00c6a","Type":"ContainerDied","Data":"f2736f2f22ecfefd3c79d1735997c8a291337f6f1bc89b649fd8f7215586fd09"}
Nov 01 07:09:13.932609 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.932527    5510 scope.go:122] "RemoveContainer" containerID="acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc"
Nov 01 07:09:13.942023 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.942008    5510 scope.go:122] "RemoveContainer" containerID="acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc"
Nov 01 07:09:13.942283 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:09:13.942265    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc\": container with ID starting with acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc not found: ID does not exist" containerID="acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc"
Nov 01 07:09:13.942334 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:13.942292    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc"} err="failed to get container status \"acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc\": rpc error: code = NotFound desc = could not find container \"acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc\": container with ID starting with acaae3d9a8a393bef8f21a655345254eb5f18fec523f3f61ca56466259b6e9cc not found: ID does not exist"
Nov 01 07:09:14.095180 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:14.095102    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"]
Nov 01 07:09:14.101314 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:14.101288    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-router-scheduler-575fd9b59b-dpllw"]
Nov 01 07:09:15.234257 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:15.234224    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" path="/var/lib/kubelet/pods/f44f7ac2-2ebd-48c3-a509-c3b332c00c6a/volumes"
Nov 01 07:09:17.545367 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:09:17.545334    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:09:47.096768 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:47.096733    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:09:47.565506 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:09:47.565461    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:09:49.559009 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:49.558969    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:09:49.559676 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:09:49.559641    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:10:05.139176 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:05.139143    5510 scope.go:122] "RemoveContainer" containerID="8f4361661ec032488f72e0b398d1292630257a141d4b86563e137848e5f5dd13"
Nov 01 07:10:05.147586 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:05.147569    5510 scope.go:122] "RemoveContainer" containerID="b003fdfdd7dd19f169e6e2697d3ad22d90953b2c9ebf031b5951d386251e02cb"
Nov 01 07:10:05.158375 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:05.158344    5510 scope.go:122] "RemoveContainer" containerID="c4ee8648b69452b557c64821d2a089966655339c94bc83a9bd93f58ceed39228"
Nov 01 07:10:14.668974 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:14.668889    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:10:17.682854 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:10:17.680020    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:10:44.165955 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.165924    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"]
Nov 01 07:10:44.166444 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.166348    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerName="main"
Nov 01 07:10:44.166444 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.166366    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerName="main"
Nov 01 07:10:44.166576 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.166533    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44f7ac2-2ebd-48c3-a509-c3b332c00c6a" containerName="main"
Nov 01 07:10:44.169040 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.169018    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.270889 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.270855    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:10:44.270889 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.270882    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:10:44.271145 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.270899    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:10:44.271145 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.270924    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:10:44.272507 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.272489    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"]
Nov 01 07:10:44.304323 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.304297    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.304450 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.304335    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbx8m\" (UniqueName: \"kubernetes.io/projected/757ba806-8452-49a6-b717-13287ca78ce8-kube-api-access-rbx8m\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.304450 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.304373    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.405983 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.405950    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.406160 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.406004    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbx8m\" (UniqueName: \"kubernetes.io/projected/757ba806-8452-49a6-b717-13287ca78ce8-kube-api-access-rbx8m\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.406160 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.406060    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.406669 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.406640    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.408548 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.408531    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.466385 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.466360    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbx8m\" (UniqueName: \"kubernetes.io/projected/757ba806-8452-49a6-b717-13287ca78ce8-kube-api-access-rbx8m\") pod \"maas-api-revoke-sub-452b6d7b-mfnzn\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.481222 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.481205    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:44.855328 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:44.855241    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"]
Nov 01 07:10:44.858629 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:10:44.858605    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757ba806_8452_49a6_b717_13287ca78ce8.slice/crio-e4d9b47ff3dbcf4a4195c07bff6428f745ac2c4735b5e0214b02914d5fce27d0 WatchSource:0}: Error finding container e4d9b47ff3dbcf4a4195c07bff6428f745ac2c4735b5e0214b02914d5fce27d0: Status 404 returned error can't find the container with id e4d9b47ff3dbcf4a4195c07bff6428f745ac2c4735b5e0214b02914d5fce27d0
Nov 01 07:10:45.407972 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:45.407934    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn" event={"ID":"757ba806-8452-49a6-b717-13287ca78ce8","Type":"ContainerStarted","Data":"1a99a340bc0d99bd3e44fa7cdcb2f8666292952bfb8b3dbbdb56bc0bae47f295"}
Nov 01 07:10:45.408387 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:45.407991    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn" event={"ID":"757ba806-8452-49a6-b717-13287ca78ce8","Type":"ContainerStarted","Data":"e4d9b47ff3dbcf4a4195c07bff6428f745ac2c4735b5e0214b02914d5fce27d0"}
Nov 01 07:10:45.510930 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:45.510899    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:10:45.550305 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:45.550258    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn" podStartSLOduration=1.55024473 podStartE2EDuration="1.55024473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:10:45.549873965 +0000 UTC m=+5082.427847376" watchObservedRunningTime="2026-11-01 07:10:45.55024473 +0000 UTC m=+5082.428218137"
Nov 01 07:10:46.430858 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:46.430184    5510 generic.go:356] "Generic (PLEG): container finished" podID="757ba806-8452-49a6-b717-13287ca78ce8" containerID="1a99a340bc0d99bd3e44fa7cdcb2f8666292952bfb8b3dbbdb56bc0bae47f295" exitCode=0
Nov 01 07:10:46.430858 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:46.430311    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn" event={"ID":"757ba806-8452-49a6-b717-13287ca78ce8","Type":"ContainerDied","Data":"1a99a340bc0d99bd3e44fa7cdcb2f8666292952bfb8b3dbbdb56bc0bae47f295"}
Nov 01 07:10:47.780220 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:10:47.777159    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:10:48.275071 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.275045    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:48.295562 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.294589    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-openshift-ca\") pod \"757ba806-8452-49a6-b717-13287ca78ce8\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") "
Nov 01 07:10:48.295562 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.294641    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-xks-ca\") pod \"757ba806-8452-49a6-b717-13287ca78ce8\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") "
Nov 01 07:10:48.295562 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.294763    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757ba806-8452-49a6-b717-13287ca78ce8-kube-api-access-rbx8m\" (UniqueName: \"kubernetes.io/projected/757ba806-8452-49a6-b717-13287ca78ce8-kube-api-access-rbx8m\") pod \"757ba806-8452-49a6-b717-13287ca78ce8\" (UID: \"757ba806-8452-49a6-b717-13287ca78ce8\") "
Nov 01 07:10:48.296328 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.296031    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-openshift-ca" pod "757ba806-8452-49a6-b717-13287ca78ce8" (UID: "757ba806-8452-49a6-b717-13287ca78ce8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:10:48.311265 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.311226    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757ba806-8452-49a6-b717-13287ca78ce8-kube-api-access-rbx8m" pod "757ba806-8452-49a6-b717-13287ca78ce8" (UID: "757ba806-8452-49a6-b717-13287ca78ce8"). InnerVolumeSpecName "kube-api-access-rbx8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:10:48.311501 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.311472    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-xks-ca" pod "757ba806-8452-49a6-b717-13287ca78ce8" (UID: "757ba806-8452-49a6-b717-13287ca78ce8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:10:48.397855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.395720    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbx8m\" (UniqueName: \"kubernetes.io/projected/757ba806-8452-49a6-b717-13287ca78ce8-kube-api-access-rbx8m\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:10:48.397855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.395755    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:10:48.397855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.395773    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ba806-8452-49a6-b717-13287ca78ce8-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:10:48.599850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.597863    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn" event={"ID":"757ba806-8452-49a6-b717-13287ca78ce8","Type":"ContainerDied","Data":"e4d9b47ff3dbcf4a4195c07bff6428f745ac2c4735b5e0214b02914d5fce27d0"}
Nov 01 07:10:48.599850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.598045    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d9b47ff3dbcf4a4195c07bff6428f745ac2c4735b5e0214b02914d5fce27d0"
Nov 01 07:10:48.599850 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:48.598330    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"
Nov 01 07:10:49.524912 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:49.524867    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:10:49.525453 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:49.524933    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:10:55.047267 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:55.047234    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"]
Nov 01 07:10:55.096888 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:55.096857    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-rr77f"]
Nov 01 07:10:55.234760 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:10:55.234727    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa8486c-e622-4b2a-bb1c-24b030da4531" path="/var/lib/kubelet/pods/8fa8486c-e622-4b2a-bb1c-24b030da4531/volumes"
Nov 01 07:11:00.779054 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:00.779009    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:11:17.544607 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:11:17.544568    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:11:17.599659 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.599628    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"]
Nov 01 07:11:17.606853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.600634    5510 cpu_manager.go:414] "containerMap: removing container" podUID="757ba806-8452-49a6-b717-13287ca78ce8" containerName="revoke-keys"
Nov 01 07:11:17.606853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.600661    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757ba806-8452-49a6-b717-13287ca78ce8" containerName="revoke-keys"
Nov 01 07:11:17.606853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.600974    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="757ba806-8452-49a6-b717-13287ca78ce8" containerName="revoke-keys"
Nov 01 07:11:17.606853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.605100    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.661554 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.661516    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"]
Nov 01 07:11:17.708642 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.708612    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:11:17.708873 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.708749    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:11:17.708873 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.708768    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:11:17.708873 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.708767    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:11:17.785283 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.785254    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmk2v\" (UniqueName: \"kubernetes.io/projected/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-kube-api-access-jmk2v\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.785452 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.785306    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.785452 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.785436    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.886672 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.886588    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.886672 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.886674    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmk2v\" (UniqueName: \"kubernetes.io/projected/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-kube-api-access-jmk2v\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.886938 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.886706    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.887321 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.887298    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.889154 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.889129    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:17.947455 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:17.947414    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmk2v\" (UniqueName: \"kubernetes.io/projected/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-kube-api-access-jmk2v\") pod \"maas-api-revoke-sub-3e31a506-gv624\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:18.219428 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:18.219400    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:18.507275 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:18.507181    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:11:18.619190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:18.619166    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"]
Nov 01 07:11:18.621310 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:11:18.621273    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe650a0_e8d5_4b95_844f_cef0158dbc1c.slice/crio-152fc9777d8d659f84428cf4a665ac01c002eace8a45b7aeb597c7e64a31b55a WatchSource:0}: Error finding container 152fc9777d8d659f84428cf4a665ac01c002eace8a45b7aeb597c7e64a31b55a: Status 404 returned error can't find the container with id 152fc9777d8d659f84428cf4a665ac01c002eace8a45b7aeb597c7e64a31b55a
Nov 01 07:11:18.760251 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:18.760170    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624" event={"ID":"ebe650a0-e8d5-4b95-844f-cef0158dbc1c","Type":"ContainerStarted","Data":"152fc9777d8d659f84428cf4a665ac01c002eace8a45b7aeb597c7e64a31b55a"}
Nov 01 07:11:19.807860 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:19.801085    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624" event={"ID":"ebe650a0-e8d5-4b95-844f-cef0158dbc1c","Type":"ContainerStarted","Data":"30ed291172ba835f2565f66eea6c62d39c954424b4f49895deb711a6dfe05f07"}
Nov 01 07:11:20.831857 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:20.831224    5510 generic.go:356] "Generic (PLEG): container finished" podID="ebe650a0-e8d5-4b95-844f-cef0158dbc1c" containerID="30ed291172ba835f2565f66eea6c62d39c954424b4f49895deb711a6dfe05f07" exitCode=0
Nov 01 07:11:20.831857 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:20.831292    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624" event={"ID":"ebe650a0-e8d5-4b95-844f-cef0158dbc1c","Type":"ContainerDied","Data":"30ed291172ba835f2565f66eea6c62d39c954424b4f49895deb711a6dfe05f07"}
Nov 01 07:11:21.972305 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:21.972274    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:22.067355 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.067324    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-kube-api-access-jmk2v\" (UniqueName: \"kubernetes.io/projected/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-kube-api-access-jmk2v\") pod \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") "
Nov 01 07:11:22.067518 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.067380    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-openshift-ca\") pod \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") "
Nov 01 07:11:22.067518 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.067470    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-xks-ca\") pod \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\" (UID: \"ebe650a0-e8d5-4b95-844f-cef0158dbc1c\") "
Nov 01 07:11:22.067747 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.067726    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-openshift-ca" pod "ebe650a0-e8d5-4b95-844f-cef0158dbc1c" (UID: "ebe650a0-e8d5-4b95-844f-cef0158dbc1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:11:22.069486 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.069464    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-kube-api-access-jmk2v" pod "ebe650a0-e8d5-4b95-844f-cef0158dbc1c" (UID: "ebe650a0-e8d5-4b95-844f-cef0158dbc1c"). InnerVolumeSpecName "kube-api-access-jmk2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:11:22.069594 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.069578    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-xks-ca" pod "ebe650a0-e8d5-4b95-844f-cef0158dbc1c" (UID: "ebe650a0-e8d5-4b95-844f-cef0158dbc1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:11:22.168931 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.168857    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:22.168931 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.168886    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:22.168931 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.168898    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmk2v\" (UniqueName: \"kubernetes.io/projected/ebe650a0-e8d5-4b95-844f-cef0158dbc1c-kube-api-access-jmk2v\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:22.841807 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.841777    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"
Nov 01 07:11:22.842014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.841817    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624" event={"ID":"ebe650a0-e8d5-4b95-844f-cef0158dbc1c","Type":"ContainerDied","Data":"152fc9777d8d659f84428cf4a665ac01c002eace8a45b7aeb597c7e64a31b55a"}
Nov 01 07:11:22.842014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:22.841906    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="152fc9777d8d659f84428cf4a665ac01c002eace8a45b7aeb597c7e64a31b55a"
Nov 01 07:11:29.045352 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:29.045317    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"]
Nov 01 07:11:29.095729 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:29.095697    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r7vjt"]
Nov 01 07:11:29.234674 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:29.234638    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9339f07a-fcde-4fd9-b3fc-1fb8e619eb20" path="/var/lib/kubelet/pods/9339f07a-fcde-4fd9-b3fc-1fb8e619eb20/volumes"
Nov 01 07:11:30.369963 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:30.369929    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:11:40.083241 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:40.083207    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:11:41.882528 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.882491    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"]
Nov 01 07:11:41.883147 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.883128    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ebe650a0-e8d5-4b95-844f-cef0158dbc1c" containerName="revoke-keys"
Nov 01 07:11:41.883204 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.883151    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe650a0-e8d5-4b95-844f-cef0158dbc1c" containerName="revoke-keys"
Nov 01 07:11:41.883274 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.883263    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe650a0-e8d5-4b95-844f-cef0158dbc1c" containerName="revoke-keys"
Nov 01 07:11:41.887700 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.887682    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:41.918806 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.918772    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spx2q\" (UniqueName: \"kubernetes.io/projected/e8b93737-4a6e-408c-a51a-de76929ffb90-kube-api-access-spx2q\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:41.919004 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.918906    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:41.919004 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.918959    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:41.939640 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.939622    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:11:41.989120 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.989074    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:11:41.989120 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.989077    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:11:41.989120 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.989108    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:11:41.990888 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:41.990865    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"]
Nov 01 07:11:42.026491 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.026463    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:42.026645 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.026507    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:42.026645 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.026584    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spx2q\" (UniqueName: \"kubernetes.io/projected/e8b93737-4a6e-408c-a51a-de76929ffb90-kube-api-access-spx2q\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:42.027328 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.027305    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:42.029082 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.029061    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:42.086987 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.086957    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spx2q\" (UniqueName: \"kubernetes.io/projected/e8b93737-4a6e-408c-a51a-de76929ffb90-kube-api-access-spx2q\") pod \"maas-api-revoke-sub-c09cd217-qc2zl\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:42.200514 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.200431    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:42.568935 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.568901    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"]
Nov 01 07:11:42.571258 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:11:42.571225    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b93737_4a6e_408c_a51a_de76929ffb90.slice/crio-3ccd595b003c049313319952f0e8edcab15d642a8c4c1b8b340c4929553c1a0d WatchSource:0}: Error finding container 3ccd595b003c049313319952f0e8edcab15d642a8c4c1b8b340c4929553c1a0d: Status 404 returned error can't find the container with id 3ccd595b003c049313319952f0e8edcab15d642a8c4c1b8b340c4929553c1a0d
Nov 01 07:11:42.995925 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.995891    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl" event={"ID":"e8b93737-4a6e-408c-a51a-de76929ffb90","Type":"ContainerStarted","Data":"b821715021afc401a0761aa1b0f5771c3dfb36a9bf9f9a798a30f9364a8151fa"}
Nov 01 07:11:42.996306 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:42.995953    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl" event={"ID":"e8b93737-4a6e-408c-a51a-de76929ffb90","Type":"ContainerStarted","Data":"3ccd595b003c049313319952f0e8edcab15d642a8c4c1b8b340c4929553c1a0d"}
Nov 01 07:11:43.112220 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:43.112163    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl" podStartSLOduration=2.112147338 podStartE2EDuration="2.112147338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:11:43.110246109 +0000 UTC m=+5139.988219520" watchObservedRunningTime="2026-11-01 07:11:43.112147338 +0000 UTC m=+5139.990120745"
Nov 01 07:11:44.003557 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:44.003525    5510 generic.go:356] "Generic (PLEG): container finished" podID="e8b93737-4a6e-408c-a51a-de76929ffb90" containerID="b821715021afc401a0761aa1b0f5771c3dfb36a9bf9f9a798a30f9364a8151fa" exitCode=0
Nov 01 07:11:44.003984 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:44.003611    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl" event={"ID":"e8b93737-4a6e-408c-a51a-de76929ffb90","Type":"ContainerDied","Data":"b821715021afc401a0761aa1b0f5771c3dfb36a9bf9f9a798a30f9364a8151fa"}
Nov 01 07:11:45.147335 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.147309    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:45.155489 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.155468    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-openshift-ca\") pod \"e8b93737-4a6e-408c-a51a-de76929ffb90\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") "
Nov 01 07:11:45.155598 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.155583    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b93737-4a6e-408c-a51a-de76929ffb90-kube-api-access-spx2q\" (UniqueName: \"kubernetes.io/projected/e8b93737-4a6e-408c-a51a-de76929ffb90-kube-api-access-spx2q\") pod \"e8b93737-4a6e-408c-a51a-de76929ffb90\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") "
Nov 01 07:11:45.155639 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.155607    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-xks-ca\") pod \"e8b93737-4a6e-408c-a51a-de76929ffb90\" (UID: \"e8b93737-4a6e-408c-a51a-de76929ffb90\") "
Nov 01 07:11:45.155878 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.155852    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-openshift-ca" pod "e8b93737-4a6e-408c-a51a-de76929ffb90" (UID: "e8b93737-4a6e-408c-a51a-de76929ffb90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:11:45.157752 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.157733    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b93737-4a6e-408c-a51a-de76929ffb90-kube-api-access-spx2q" pod "e8b93737-4a6e-408c-a51a-de76929ffb90" (UID: "e8b93737-4a6e-408c-a51a-de76929ffb90"). InnerVolumeSpecName "kube-api-access-spx2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:11:45.157881 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.157865    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-xks-ca" pod "e8b93737-4a6e-408c-a51a-de76929ffb90" (UID: "e8b93737-4a6e-408c-a51a-de76929ffb90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:11:45.257168 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.257142    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:45.257168 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.257166    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spx2q\" (UniqueName: \"kubernetes.io/projected/e8b93737-4a6e-408c-a51a-de76929ffb90-kube-api-access-spx2q\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:45.257342 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:45.257175    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b93737-4a6e-408c-a51a-de76929ffb90-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:46.014537 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:46.014507    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"
Nov 01 07:11:46.014537 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:46.014532    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl" event={"ID":"e8b93737-4a6e-408c-a51a-de76929ffb90","Type":"ContainerDied","Data":"3ccd595b003c049313319952f0e8edcab15d642a8c4c1b8b340c4929553c1a0d"}
Nov 01 07:11:46.014750 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:46.014575    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ccd595b003c049313319952f0e8edcab15d642a8c4c1b8b340c4929553c1a0d"
Nov 01 07:11:47.544160 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:11:47.544123    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:11:54.695072 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:54.695037    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:11:57.044331 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:57.044295    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"]
Nov 01 07:11:57.093517 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:57.093487    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5gwz"]
Nov 01 07:11:57.235091 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:11:57.235055    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea8d3ba-c60a-478e-bab4-48ac542c3df7" path="/var/lib/kubelet/pods/1ea8d3ba-c60a-478e-bab4-48ac542c3df7/volumes"
Nov 01 07:12:05.195995 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:05.195965    5510 scope.go:122] "RemoveContainer" containerID="6afbdcedf87e146250f3a6ff98900c7689c8720f86380dd6f59123e3b823fd3d"
Nov 01 07:12:07.637625 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:07.637586    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"]
Nov 01 07:12:10.061330 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:10.061295    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"]
Nov 01 07:12:10.110497 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:10.110459    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xlppn"]
Nov 01 07:12:11.234812 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:11.234775    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13525afa-0790-49a6-94f9-f13ba73074f3" path="/var/lib/kubelet/pods/13525afa-0790-49a6-94f9-f13ba73074f3/volumes"
Nov 01 07:12:12.638105 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:12.638049    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd" podUID="a714429e-0dca-453f-bb85-d62bef7ad6ad" containerName="payload-pre-processing" containerID="cri-o://4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78" gracePeriod=25
Nov 01 07:12:12.878745 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:12.878714    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 07:12:13.016147 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.016108    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a714429e-0dca-453f-bb85-d62bef7ad6ad-kube-api-access-8v9r6\" (UniqueName: \"kubernetes.io/projected/a714429e-0dca-453f-bb85-d62bef7ad6ad-kube-api-access-8v9r6\") pod \"a714429e-0dca-453f-bb85-d62bef7ad6ad\" (UID: \"a714429e-0dca-453f-bb85-d62bef7ad6ad\") "
Nov 01 07:12:13.016350 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.016199    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a714429e-0dca-453f-bb85-d62bef7ad6ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a714429e-0dca-453f-bb85-d62bef7ad6ad-plugins-config-volume\") pod \"a714429e-0dca-453f-bb85-d62bef7ad6ad\" (UID: \"a714429e-0dca-453f-bb85-d62bef7ad6ad\") "
Nov 01 07:12:13.016596 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.016574    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a714429e-0dca-453f-bb85-d62bef7ad6ad-plugins-config-volume" pod "a714429e-0dca-453f-bb85-d62bef7ad6ad" (UID: "a714429e-0dca-453f-bb85-d62bef7ad6ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:12:13.018393 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.018370    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a714429e-0dca-453f-bb85-d62bef7ad6ad-kube-api-access-8v9r6" pod "a714429e-0dca-453f-bb85-d62bef7ad6ad" (UID: "a714429e-0dca-453f-bb85-d62bef7ad6ad"). InnerVolumeSpecName "kube-api-access-8v9r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:12:13.118012 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.117977    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a714429e-0dca-453f-bb85-d62bef7ad6ad-plugins-config-volume\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:12:13.118012 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.118008    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v9r6\" (UniqueName: \"kubernetes.io/projected/a714429e-0dca-453f-bb85-d62bef7ad6ad-kube-api-access-8v9r6\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:12:13.146316 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.146280    5510 generic.go:356] "Generic (PLEG): container finished" podID="a714429e-0dca-453f-bb85-d62bef7ad6ad" containerID="4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78" exitCode=0
Nov 01 07:12:13.146480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.146331    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"
Nov 01 07:12:13.146480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.146362    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd" event={"ID":"a714429e-0dca-453f-bb85-d62bef7ad6ad","Type":"ContainerDied","Data":"4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78"}
Nov 01 07:12:13.146480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.146419    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd" event={"ID":"a714429e-0dca-453f-bb85-d62bef7ad6ad","Type":"ContainerDied","Data":"d50bacb0ff7ac1e7c66840f5e9a6bacfe3251ba59863d3d74080b8b1ce8f741b"}
Nov 01 07:12:13.146480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.146440    5510 scope.go:122] "RemoveContainer" containerID="4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78"
Nov 01 07:12:13.155647 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.155630    5510 scope.go:122] "RemoveContainer" containerID="4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78"
Nov 01 07:12:13.155942 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:12:13.155922    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78\": container with ID starting with 4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78 not found: ID does not exist" containerID="4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78"
Nov 01 07:12:13.156008 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.155956    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78"} err="failed to get container status \"4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78\": rpc error: code = NotFound desc = could not find container \"4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78\": container with ID starting with 4fb76f60ba8906fe8fcfba2c5c17820ec8dbe5869baaaa608eb5b5068d85bd78 not found: ID does not exist"
Nov 01 07:12:13.309574 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.309496    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"]
Nov 01 07:12:13.314028 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:13.313999    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dzmqd"]
Nov 01 07:12:14.274957 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:14.274922    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal" podUID="784c45e8dd43338b3f3bbbd833ce9232" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:12:15.234307 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:15.234265    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a714429e-0dca-453f-bb85-d62bef7ad6ad" path="/var/lib/kubelet/pods/a714429e-0dca-453f-bb85-d62bef7ad6ad/volumes"
Nov 01 07:12:17.545342 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:12:17.545305    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:12:23.848266 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.848232    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-68ccf99fdf-bbf6f"]
Nov 01 07:12:23.848663 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.848635    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e8b93737-4a6e-408c-a51a-de76929ffb90" containerName="revoke-keys"
Nov 01 07:12:23.848663 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.848647    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b93737-4a6e-408c-a51a-de76929ffb90" containerName="revoke-keys"
Nov 01 07:12:23.848741 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.848673    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a714429e-0dca-453f-bb85-d62bef7ad6ad" containerName="payload-pre-processing"
Nov 01 07:12:23.848741 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.848678    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a714429e-0dca-453f-bb85-d62bef7ad6ad" containerName="payload-pre-processing"
Nov 01 07:12:23.848811 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.848730    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a714429e-0dca-453f-bb85-d62bef7ad6ad" containerName="payload-pre-processing"
Nov 01 07:12:23.848811 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.848776    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b93737-4a6e-408c-a51a-de76929ffb90" containerName="revoke-keys"
Nov 01 07:12:23.851731 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.851716    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:23.903562 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.903536    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 07:12:23.953051 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.953027    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7d8fb\""
Nov 01 07:12:23.954702 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:23.954675    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-68ccf99fdf-bbf6f"]
Nov 01 07:12:24.009493 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.009466    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"]
Nov 01 07:12:24.013005 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.012987    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:24.020808 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.020785    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f6834-d1a0-415b-a8fe-def794867e99-plugins-config-volume\") pod \"payload-processing-68ccf99fdf-bbf6f\" (UID: \"8e9f6834-d1a0-415b-a8fe-def794867e99\") " pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:24.020958 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.020937    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgplz\" (UniqueName: \"kubernetes.io/projected/8e9f6834-d1a0-415b-a8fe-def794867e99-kube-api-access-xgplz\") pod \"payload-processing-68ccf99fdf-bbf6f\" (UID: \"8e9f6834-d1a0-415b-a8fe-def794867e99\") " pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:24.071967 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.071934    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"]
Nov 01 07:12:24.122255 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.122168    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgplz\" (UniqueName: \"kubernetes.io/projected/8e9f6834-d1a0-415b-a8fe-def794867e99-kube-api-access-xgplz\") pod \"payload-processing-68ccf99fdf-bbf6f\" (UID: \"8e9f6834-d1a0-415b-a8fe-def794867e99\") " pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:24.122255 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.122243    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be361928-05c7-4596-879d-d2cfe79eabcf-plugins-config-volume\") pod \"payload-pre-processing-644bf9c6bb-8d6wr\" (UID: \"be361928-05c7-4596-879d-d2cfe79eabcf\") " pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:24.122508 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.122292    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f6834-d1a0-415b-a8fe-def794867e99-plugins-config-volume\") pod \"payload-processing-68ccf99fdf-bbf6f\" (UID: \"8e9f6834-d1a0-415b-a8fe-def794867e99\") " pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:24.122508 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.122384    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bfg\" (UniqueName: \"kubernetes.io/projected/be361928-05c7-4596-879d-d2cfe79eabcf-kube-api-access-62bfg\") pod \"payload-pre-processing-644bf9c6bb-8d6wr\" (UID: \"be361928-05c7-4596-879d-d2cfe79eabcf\") " pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:24.123101 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.123081    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f6834-d1a0-415b-a8fe-def794867e99-plugins-config-volume\") pod \"payload-processing-68ccf99fdf-bbf6f\" (UID: \"8e9f6834-d1a0-415b-a8fe-def794867e99\") " pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:24.183468 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.183439    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgplz\" (UniqueName: \"kubernetes.io/projected/8e9f6834-d1a0-415b-a8fe-def794867e99-kube-api-access-xgplz\") pod \"payload-processing-68ccf99fdf-bbf6f\" (UID: \"8e9f6834-d1a0-415b-a8fe-def794867e99\") " pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:24.223808 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.223778    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be361928-05c7-4596-879d-d2cfe79eabcf-plugins-config-volume\") pod \"payload-pre-processing-644bf9c6bb-8d6wr\" (UID: \"be361928-05c7-4596-879d-d2cfe79eabcf\") " pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:24.224006 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.223866    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bfg\" (UniqueName: \"kubernetes.io/projected/be361928-05c7-4596-879d-d2cfe79eabcf-kube-api-access-62bfg\") pod \"payload-pre-processing-644bf9c6bb-8d6wr\" (UID: \"be361928-05c7-4596-879d-d2cfe79eabcf\") " pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:24.224496 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.224470    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be361928-05c7-4596-879d-d2cfe79eabcf-plugins-config-volume\") pod \"payload-pre-processing-644bf9c6bb-8d6wr\" (UID: \"be361928-05c7-4596-879d-d2cfe79eabcf\") " pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:24.284622 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.284597    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bfg\" (UniqueName: \"kubernetes.io/projected/be361928-05c7-4596-879d-d2cfe79eabcf-kube-api-access-62bfg\") pod \"payload-pre-processing-644bf9c6bb-8d6wr\" (UID: \"be361928-05c7-4596-879d-d2cfe79eabcf\") " pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:24.325963 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.325937    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:24.463915 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.463884    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:24.695966 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.695880    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"]
Nov 01 07:12:24.699238 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:12:24.699208    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe361928_05c7_4596_879d_d2cfe79eabcf.slice/crio-62acb18c8c96e6accbbaacd8bf9f17d95ac95a46013b44381e1452acc057aa50 WatchSource:0}: Error finding container 62acb18c8c96e6accbbaacd8bf9f17d95ac95a46013b44381e1452acc057aa50: Status 404 returned error can't find the container with id 62acb18c8c96e6accbbaacd8bf9f17d95ac95a46013b44381e1452acc057aa50
Nov 01 07:12:24.834737 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:24.834662    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-68ccf99fdf-bbf6f"]
Nov 01 07:12:24.837076 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:12:24.837050    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9f6834_d1a0_415b_a8fe_def794867e99.slice/crio-009d2d1baf4ec1b838b45192e732e6bb6f34c7dc2dc7d391a124653d7b5a83b2 WatchSource:0}: Error finding container 009d2d1baf4ec1b838b45192e732e6bb6f34c7dc2dc7d391a124653d7b5a83b2: Status 404 returned error can't find the container with id 009d2d1baf4ec1b838b45192e732e6bb6f34c7dc2dc7d391a124653d7b5a83b2
Nov 01 07:12:25.204709 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:25.204681    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr" event={"ID":"be361928-05c7-4596-879d-d2cfe79eabcf","Type":"ContainerStarted","Data":"fc198f6b606d04390fe0a3051986a9ca251ccbdf89fec5f23b0f6a5c603dbbbb"}
Nov 01 07:12:25.205184 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:25.204729    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr" event={"ID":"be361928-05c7-4596-879d-d2cfe79eabcf","Type":"ContainerStarted","Data":"62acb18c8c96e6accbbaacd8bf9f17d95ac95a46013b44381e1452acc057aa50"}
Nov 01 07:12:25.205184 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:25.204801    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:25.206354 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:25.206335    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f" event={"ID":"8e9f6834-d1a0-415b-a8fe-def794867e99","Type":"ContainerStarted","Data":"12c25bdcd423d6f9e74313451d107077b5eec1b362e610d6f268b22bc54be053"}
Nov 01 07:12:25.206421 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:25.206371    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f" event={"ID":"8e9f6834-d1a0-415b-a8fe-def794867e99","Type":"ContainerStarted","Data":"009d2d1baf4ec1b838b45192e732e6bb6f34c7dc2dc7d391a124653d7b5a83b2"}
Nov 01 07:12:25.206421 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:25.206406    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:25.320259 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:25.320203    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr" podStartSLOduration=2.32018833 podStartE2EDuration="2.32018833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:12:25.319163914 +0000 UTC m=+5182.197137315" watchObservedRunningTime="2026-11-01 07:12:25.32018833 +0000 UTC m=+5182.198161740"
Nov 01 07:12:25.433670 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:25.433614    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f" podStartSLOduration=2.4335787 podStartE2EDuration="2.4335787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:12:25.433560411 +0000 UTC m=+5182.311533821" watchObservedRunningTime="2026-11-01 07:12:25.4335787 +0000 UTC m=+5182.311552113"
Nov 01 07:12:36.214054 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:36.214018    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-68ccf99fdf-bbf6f"
Nov 01 07:12:36.214543 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:36.214185    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-644bf9c6bb-8d6wr"
Nov 01 07:12:47.544932 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:12:47.544895    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:12:53.799052 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:12:53.799016    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:13:05.216157 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:13:05.216122    5510 scope.go:122] "RemoveContainer" containerID="f252aa7affee3201c9f787b92b925d9027b1f9c789708f577b45932c877e0724"
Nov 01 07:13:05.224650 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:13:05.224631    5510 scope.go:122] "RemoveContainer" containerID="f92acedca10a75c28a5c30227a5bdab272142e2170bb0f33b137e22a3db596b2"
Nov 01 07:13:17.545013 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:13:17.544926    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:13:47.544241 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:13:47.544200    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:14:05.259245 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:05.259215    5510 scope.go:122] "RemoveContainer" containerID="309cea23fa4863867e2974018e40143d318994b23cad41cb0a50894710393d87"
Nov 01 07:14:17.544668 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:14:17.544623    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:14:33.685839 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.685802    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"]
Nov 01 07:14:33.689649 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.689632    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.791126 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.791084    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:14:33.791126 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.791099    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:14:33.791126 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.791122    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:14:33.791433 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.791294    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:14:33.792963 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.792940    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"]
Nov 01 07:14:33.813260 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.813231    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.813417 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.813343    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b6wz\" (UniqueName: \"kubernetes.io/projected/8d7c5e52-80b2-470a-9293-3e267fea972a-kube-api-access-6b6wz\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.813470 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.813413    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.914802 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.914770    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.914972 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.914877    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b6wz\" (UniqueName: \"kubernetes.io/projected/8d7c5e52-80b2-470a-9293-3e267fea972a-kube-api-access-6b6wz\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.914972 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.914936    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.915446 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.915423    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.917323 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.917295    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:33.975771 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:33.975740    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b6wz\" (UniqueName: \"kubernetes.io/projected/8d7c5e52-80b2-470a-9293-3e267fea972a-kube-api-access-6b6wz\") pod \"maas-api-revoke-sub-e2547622-vpmz6\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:34.002454 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:34.002432    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:34.376119 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:34.376096    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"]
Nov 01 07:14:34.379161 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:14:34.379121    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7c5e52_80b2_470a_9293_3e267fea972a.slice/crio-7505c5ff3a6598b2b5588d02c34359994d0578db4b87a703e168ef0e106026ee WatchSource:0}: Error finding container 7505c5ff3a6598b2b5588d02c34359994d0578db4b87a703e168ef0e106026ee: Status 404 returned error can't find the container with id 7505c5ff3a6598b2b5588d02c34359994d0578db4b87a703e168ef0e106026ee
Nov 01 07:14:34.837145 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:34.837116    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6" event={"ID":"8d7c5e52-80b2-470a-9293-3e267fea972a","Type":"ContainerStarted","Data":"a3fb9f5aaaf41f8ebc20c25efa39fb0f6d413d0b850fba20cfb86c18f4e44691"}
Nov 01 07:14:34.837522 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:34.837168    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6" event={"ID":"8d7c5e52-80b2-470a-9293-3e267fea972a","Type":"ContainerStarted","Data":"7505c5ff3a6598b2b5588d02c34359994d0578db4b87a703e168ef0e106026ee"}
Nov 01 07:14:34.952860 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:34.952741    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6" podStartSLOduration=1.952726881 podStartE2EDuration="1.952726881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:14:34.951674362 +0000 UTC m=+5311.829647786" watchObservedRunningTime="2026-11-01 07:14:34.952726881 +0000 UTC m=+5311.830700337"
Nov 01 07:14:35.849674 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:35.849642    5510 generic.go:356] "Generic (PLEG): container finished" podID="8d7c5e52-80b2-470a-9293-3e267fea972a" containerID="a3fb9f5aaaf41f8ebc20c25efa39fb0f6d413d0b850fba20cfb86c18f4e44691" exitCode=0
Nov 01 07:14:35.850079 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:35.849724    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6" event={"ID":"8d7c5e52-80b2-470a-9293-3e267fea972a","Type":"ContainerDied","Data":"a3fb9f5aaaf41f8ebc20c25efa39fb0f6d413d0b850fba20cfb86c18f4e44691"}
Nov 01 07:14:36.987986 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:36.987957    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:37.148390 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.148321    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-xks-ca\") pod \"8d7c5e52-80b2-470a-9293-3e267fea972a\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") "
Nov 01 07:14:37.148390 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.148367    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-openshift-ca\") pod \"8d7c5e52-80b2-470a-9293-3e267fea972a\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") "
Nov 01 07:14:37.148622 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.148425    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7c5e52-80b2-470a-9293-3e267fea972a-kube-api-access-6b6wz\" (UniqueName: \"kubernetes.io/projected/8d7c5e52-80b2-470a-9293-3e267fea972a-kube-api-access-6b6wz\") pod \"8d7c5e52-80b2-470a-9293-3e267fea972a\" (UID: \"8d7c5e52-80b2-470a-9293-3e267fea972a\") "
Nov 01 07:14:37.148801 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.148779    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-openshift-ca" pod "8d7c5e52-80b2-470a-9293-3e267fea972a" (UID: "8d7c5e52-80b2-470a-9293-3e267fea972a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:14:37.150658 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.150625    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-xks-ca" pod "8d7c5e52-80b2-470a-9293-3e267fea972a" (UID: "8d7c5e52-80b2-470a-9293-3e267fea972a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:14:37.150773 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.150708    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7c5e52-80b2-470a-9293-3e267fea972a-kube-api-access-6b6wz" pod "8d7c5e52-80b2-470a-9293-3e267fea972a" (UID: "8d7c5e52-80b2-470a-9293-3e267fea972a"). InnerVolumeSpecName "kube-api-access-6b6wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:14:37.250020 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.249992    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:37.250020 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.250017    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7c5e52-80b2-470a-9293-3e267fea972a-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:37.250175 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.250028    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b6wz\" (UniqueName: \"kubernetes.io/projected/8d7c5e52-80b2-470a-9293-3e267fea972a-kube-api-access-6b6wz\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:37.860846 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.860797    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"
Nov 01 07:14:37.861025 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.860854    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6" event={"ID":"8d7c5e52-80b2-470a-9293-3e267fea972a","Type":"ContainerDied","Data":"7505c5ff3a6598b2b5588d02c34359994d0578db4b87a703e168ef0e106026ee"}
Nov 01 07:14:37.861025 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:37.860902    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7505c5ff3a6598b2b5588d02c34359994d0578db4b87a703e168ef0e106026ee"
Nov 01 07:14:47.545132 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:14:47.545051    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:14:48.285409 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:48.285380    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:14:58.558434 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:14:58.558395    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:15:00.158894 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.155906    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"]
Nov 01 07:15:00.158894 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.156537    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8d7c5e52-80b2-470a-9293-3e267fea972a" containerName="revoke-keys"
Nov 01 07:15:00.158894 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.156556    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7c5e52-80b2-470a-9293-3e267fea972a" containerName="revoke-keys"
Nov 01 07:15:00.158894 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.156690    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7c5e52-80b2-470a-9293-3e267fea972a" containerName="revoke-keys"
Nov 01 07:15:00.178190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.178161    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:00.245043 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.245014    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"]
Nov 01 07:15:00.307854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.281114    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:15:00.307854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.281288    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:15:00.307854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.281982    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:15:00.341855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.325347    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891955-2rvgq\" (UID: \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:00.341855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.325445    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fbkz\" (UniqueName: \"kubernetes.io/projected/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-kube-api-access-5fbkz\") pod \"maas-api-key-cleanup-29891955-2rvgq\" (UID: \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:00.446699 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.445329    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891955-2rvgq\" (UID: \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:00.446699 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.445402    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fbkz\" (UniqueName: \"kubernetes.io/projected/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-kube-api-access-5fbkz\") pod \"maas-api-key-cleanup-29891955-2rvgq\" (UID: \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:00.446699 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.446597    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891955-2rvgq\" (UID: \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:00.566882 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.556894    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fbkz\" (UniqueName: \"kubernetes.io/projected/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-kube-api-access-5fbkz\") pod \"maas-api-key-cleanup-29891955-2rvgq\" (UID: \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:00.830231 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:00.830201    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:01.745096 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:01.745060    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"]
Nov 01 07:15:01.748120 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:15:01.748091    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1f50a9_c683_4bd2_a079_245d60f4c5d4.slice/crio-72426dde68bc789413a66c0bc0655d76afdd6c71a63906c082d769dfc9456f2c WatchSource:0}: Error finding container 72426dde68bc789413a66c0bc0655d76afdd6c71a63906c082d769dfc9456f2c: Status 404 returned error can't find the container with id 72426dde68bc789413a66c0bc0655d76afdd6c71a63906c082d769dfc9456f2c
Nov 01 07:15:02.017451 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:02.017365    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq" event={"ID":"bc1f50a9-c683-4bd2-a079-245d60f4c5d4","Type":"ContainerStarted","Data":"72426dde68bc789413a66c0bc0655d76afdd6c71a63906c082d769dfc9456f2c"}
Nov 01 07:15:03.023853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:03.023794    5510 generic.go:356] "Generic (PLEG): container finished" podID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerID="e5baed94b95b388c57fd85a6e7413a2b25c0a0185b30486f5fbb80f8573c5d0e" exitCode=60
Nov 01 07:15:03.024278 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:03.023883    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq" event={"ID":"bc1f50a9-c683-4bd2-a079-245d60f4c5d4","Type":"ContainerDied","Data":"e5baed94b95b388c57fd85a6e7413a2b25c0a0185b30486f5fbb80f8573c5d0e"}
Nov 01 07:15:03.024278 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:03.024192    5510 scope.go:122] "RemoveContainer" containerID="e5baed94b95b388c57fd85a6e7413a2b25c0a0185b30486f5fbb80f8573c5d0e"
Nov 01 07:15:04.030435 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:04.030343    5510 generic.go:356] "Generic (PLEG): container finished" podID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerID="5aa70af53eec4d542f1ce63403f0e29f72a5871a4f5c450de52247da718ae31a" exitCode=60
Nov 01 07:15:04.030435 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:04.030428    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq" event={"ID":"bc1f50a9-c683-4bd2-a079-245d60f4c5d4","Type":"ContainerDied","Data":"5aa70af53eec4d542f1ce63403f0e29f72a5871a4f5c450de52247da718ae31a"}
Nov 01 07:15:04.030938 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:04.030480    5510 scope.go:122] "RemoveContainer" containerID="e5baed94b95b388c57fd85a6e7413a2b25c0a0185b30486f5fbb80f8573c5d0e"
Nov 01 07:15:04.030938 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:04.030804    5510 scope.go:122] "RemoveContainer" containerID="5aa70af53eec4d542f1ce63403f0e29f72a5871a4f5c450de52247da718ae31a"
Nov 01 07:15:04.031167 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:15:04.031141    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891955-2rvgq_odh-ai-gateway-infra(bc1f50a9-c683-4bd2-a079-245d60f4c5d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4"
Nov 01 07:15:05.037456 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:05.037422    5510 scope.go:122] "RemoveContainer" containerID="5aa70af53eec4d542f1ce63403f0e29f72a5871a4f5c450de52247da718ae31a"
Nov 01 07:15:05.038032 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:15:05.037715    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891955-2rvgq_odh-ai-gateway-infra(bc1f50a9-c683-4bd2-a079-245d60f4c5d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4"
Nov 01 07:15:14.228948 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:14.228910    5510 scope.go:122] "RemoveContainer" containerID="5aa70af53eec4d542f1ce63403f0e29f72a5871a4f5c450de52247da718ae31a"
Nov 01 07:15:15.085713 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:15.085681    5510 generic.go:356] "Generic (PLEG): container finished" podID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerID="9cbe46aa1ab07e39f1859a1f17f1aebc0910ddd7cd75425821ff21ce7b2532f7" exitCode=60
Nov 01 07:15:15.085921 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:15.085758    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq" event={"ID":"bc1f50a9-c683-4bd2-a079-245d60f4c5d4","Type":"ContainerDied","Data":"9cbe46aa1ab07e39f1859a1f17f1aebc0910ddd7cd75425821ff21ce7b2532f7"}
Nov 01 07:15:15.085921 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:15.085868    5510 scope.go:122] "RemoveContainer" containerID="5aa70af53eec4d542f1ce63403f0e29f72a5871a4f5c450de52247da718ae31a"
Nov 01 07:15:15.086223 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:15.086204    5510 scope.go:122] "RemoveContainer" containerID="9cbe46aa1ab07e39f1859a1f17f1aebc0910ddd7cd75425821ff21ce7b2532f7"
Nov 01 07:15:15.086552 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:15:15.086449    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891955-2rvgq_odh-ai-gateway-infra(bc1f50a9-c683-4bd2-a079-245d60f4c5d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4"
Nov 01 07:15:16.210373 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:16.210332    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"]
Nov 01 07:15:16.352691 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:16.352667    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:16.432027 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:16.431992    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-maas-api-service-ca\") pod \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\" (UID: \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\") "
Nov 01 07:15:16.432235 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:16.432050    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-kube-api-access-5fbkz\" (UniqueName: \"kubernetes.io/projected/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-kube-api-access-5fbkz\") pod \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\" (UID: \"bc1f50a9-c683-4bd2-a079-245d60f4c5d4\") "
Nov 01 07:15:16.432383 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:16.432363    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-maas-api-service-ca" pod "bc1f50a9-c683-4bd2-a079-245d60f4c5d4" (UID: "bc1f50a9-c683-4bd2-a079-245d60f4c5d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:15:16.434215 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:16.434196    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-kube-api-access-5fbkz" pod "bc1f50a9-c683-4bd2-a079-245d60f4c5d4" (UID: "bc1f50a9-c683-4bd2-a079-245d60f4c5d4"). InnerVolumeSpecName "kube-api-access-5fbkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:16.533129 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:16.533045    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-maas-api-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:16.533129 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:16.533075    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fbkz\" (UniqueName: \"kubernetes.io/projected/bc1f50a9-c683-4bd2-a079-245d60f4c5d4-kube-api-access-5fbkz\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:17.099479 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:17.099448    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"
Nov 01 07:15:17.099479 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:17.099473    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq" event={"ID":"bc1f50a9-c683-4bd2-a079-245d60f4c5d4","Type":"ContainerDied","Data":"72426dde68bc789413a66c0bc0655d76afdd6c71a63906c082d769dfc9456f2c"}
Nov 01 07:15:17.099758 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:17.099533    5510 scope.go:122] "RemoveContainer" containerID="9cbe46aa1ab07e39f1859a1f17f1aebc0910ddd7cd75425821ff21ce7b2532f7"
Nov 01 07:15:17.262846 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:17.262797    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"]
Nov 01 07:15:17.311938 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:17.311909    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-2rvgq"]
Nov 01 07:15:17.545507 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:15:17.545471    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:15:19.234106 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:19.234072    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" path="/var/lib/kubelet/pods/bc1f50a9-c683-4bd2-a079-245d60f4c5d4/volumes"
Nov 01 07:15:26.797691 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.797658    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"]
Nov 01 07:15:26.798872 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.798819    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:15:26.799014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.798875    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:15:26.799014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.798928    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:15:26.799014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.798937    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:15:26.799190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.799145    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:15:26.799190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.799166    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:15:26.799190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.799176    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:15:26.807094 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.807060    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:26.860293 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.860257    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"]
Nov 01 07:15:26.906854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.906796    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:15:26.907023 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.906849    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:15:26.907023 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.906895    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:15:26.907023 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.906921    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:15:26.931950 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.931919    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:26.932213 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.932077    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f64df\" (UniqueName: \"kubernetes.io/projected/d74ab553-ac6b-49e6-913a-77fa38cc49a1-kube-api-access-f64df\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:26.932213 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:26.932154    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:27.033268 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:27.033236    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64df\" (UniqueName: \"kubernetes.io/projected/d74ab553-ac6b-49e6-913a-77fa38cc49a1-kube-api-access-f64df\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:27.033480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:27.033292    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:27.033480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:27.033334    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:27.034048 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:27.034023    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:27.035812 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:27.035795    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:27.093789 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:27.093724    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f64df\" (UniqueName: \"kubernetes.io/projected/d74ab553-ac6b-49e6-913a-77fa38cc49a1-kube-api-access-f64df\") pod \"maas-api-revoke-sub-ec843018-zk2qb\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:27.125738 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:27.125703    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:27.497462 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:27.497437    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"]
Nov 01 07:15:27.499513 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:15:27.499484    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74ab553_ac6b_49e6_913a_77fa38cc49a1.slice/crio-e2fdcb604cb1ce65edfcaaec0cca9181d559e7834481794b6eccc2f36d031769 WatchSource:0}: Error finding container e2fdcb604cb1ce65edfcaaec0cca9181d559e7834481794b6eccc2f36d031769: Status 404 returned error can't find the container with id e2fdcb604cb1ce65edfcaaec0cca9181d559e7834481794b6eccc2f36d031769
Nov 01 07:15:28.152483 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:28.152394    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb" event={"ID":"d74ab553-ac6b-49e6-913a-77fa38cc49a1","Type":"ContainerStarted","Data":"2a656600f5feafc46912b79c6370760c4542b5988008b14d537e88ce0d732a61"}
Nov 01 07:15:28.152483 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:28.152452    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb" event={"ID":"d74ab553-ac6b-49e6-913a-77fa38cc49a1","Type":"ContainerStarted","Data":"e2fdcb604cb1ce65edfcaaec0cca9181d559e7834481794b6eccc2f36d031769"}
Nov 01 07:15:28.268087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:28.268037    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb" podStartSLOduration=2.2680227410000002 podStartE2EDuration="2.268022741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:15:28.266497734 +0000 UTC m=+5365.144471145" watchObservedRunningTime="2026-11-01 07:15:28.268022741 +0000 UTC m=+5365.145996152"
Nov 01 07:15:29.158988 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:29.158953    5510 generic.go:356] "Generic (PLEG): container finished" podID="d74ab553-ac6b-49e6-913a-77fa38cc49a1" containerID="2a656600f5feafc46912b79c6370760c4542b5988008b14d537e88ce0d732a61" exitCode=0
Nov 01 07:15:29.159401 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:29.159042    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb" event={"ID":"d74ab553-ac6b-49e6-913a-77fa38cc49a1","Type":"ContainerDied","Data":"2a656600f5feafc46912b79c6370760c4542b5988008b14d537e88ce0d732a61"}
Nov 01 07:15:30.301569 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.301545    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:30.466551 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.466524    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-xks-ca\") pod \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") "
Nov 01 07:15:30.466731 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.466562    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74ab553-ac6b-49e6-913a-77fa38cc49a1-kube-api-access-f64df\" (UniqueName: \"kubernetes.io/projected/d74ab553-ac6b-49e6-913a-77fa38cc49a1-kube-api-access-f64df\") pod \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") "
Nov 01 07:15:30.466731 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.466683    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-openshift-ca\") pod \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\" (UID: \"d74ab553-ac6b-49e6-913a-77fa38cc49a1\") "
Nov 01 07:15:30.467077 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.467049    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-openshift-ca" pod "d74ab553-ac6b-49e6-913a-77fa38cc49a1" (UID: "d74ab553-ac6b-49e6-913a-77fa38cc49a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:15:30.468804 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.468784    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-xks-ca" pod "d74ab553-ac6b-49e6-913a-77fa38cc49a1" (UID: "d74ab553-ac6b-49e6-913a-77fa38cc49a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:15:30.468911 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.468875    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74ab553-ac6b-49e6-913a-77fa38cc49a1-kube-api-access-f64df" pod "d74ab553-ac6b-49e6-913a-77fa38cc49a1" (UID: "d74ab553-ac6b-49e6-913a-77fa38cc49a1"). InnerVolumeSpecName "kube-api-access-f64df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:30.568165 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.568136    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:30.568165 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.568161    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f64df\" (UniqueName: \"kubernetes.io/projected/d74ab553-ac6b-49e6-913a-77fa38cc49a1-kube-api-access-f64df\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:30.568165 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.568171    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74ab553-ac6b-49e6-913a-77fa38cc49a1-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:30.689916 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:30.689885    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:15:31.169985 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:31.169949    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb" event={"ID":"d74ab553-ac6b-49e6-913a-77fa38cc49a1","Type":"ContainerDied","Data":"e2fdcb604cb1ce65edfcaaec0cca9181d559e7834481794b6eccc2f36d031769"}
Nov 01 07:15:31.170169 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:31.170008    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2fdcb604cb1ce65edfcaaec0cca9181d559e7834481794b6eccc2f36d031769"
Nov 01 07:15:31.170169 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:31.169962    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"
Nov 01 07:15:47.544022 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:15:47.543977    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:15:48.045679 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:48.045643    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"]
Nov 01 07:15:48.094848 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:48.094794    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mfnzn"]
Nov 01 07:15:49.234452 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:49.234422    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757ba806-8452-49a6-b717-13287ca78ce8" path="/var/lib/kubelet/pods/757ba806-8452-49a6-b717-13287ca78ce8/volumes"
Nov 01 07:15:53.865103 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:15:53.865069    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:16:15.547255 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.547183    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"]
Nov 01 07:16:15.547637 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.547624    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d74ab553-ac6b-49e6-913a-77fa38cc49a1" containerName="revoke-keys"
Nov 01 07:16:15.547684 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.547640    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74ab553-ac6b-49e6-913a-77fa38cc49a1" containerName="revoke-keys"
Nov 01 07:16:15.547684 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.547667    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:16:15.547684 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.547672    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1f50a9-c683-4bd2-a079-245d60f4c5d4" containerName="cleanup"
Nov 01 07:16:15.547783 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.547727    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74ab553-ac6b-49e6-913a-77fa38cc49a1" containerName="revoke-keys"
Nov 01 07:16:15.552229 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.552212    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.604222 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.604199    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:16:15.634468 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.634432    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.634609 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.634591    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6w6t\" (UniqueName: \"kubernetes.io/projected/9f813809-6f25-43fe-bddd-e0d7ceca00e1-kube-api-access-h6w6t\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.634659 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.634619    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.653958 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.653927    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:16:15.653958 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.653935    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:16:15.655710 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.655690    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"]
Nov 01 07:16:15.704680 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.704654    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:16:15.736087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.736054    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.736239 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.736138    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6w6t\" (UniqueName: \"kubernetes.io/projected/9f813809-6f25-43fe-bddd-e0d7ceca00e1-kube-api-access-h6w6t\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.736239 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.736158    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.736746 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.736728    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.738684 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.738662    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.796631 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.796606    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6w6t\" (UniqueName: \"kubernetes.io/projected/9f813809-6f25-43fe-bddd-e0d7ceca00e1-kube-api-access-h6w6t\") pod \"maas-api-revoke-sub-046e1343-f47c9\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:15.864795 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:15.864731    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:16.235031 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:16.235003    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"]
Nov 01 07:16:16.238154 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:16:16.238117    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f813809_6f25_43fe_bddd_e0d7ceca00e1.slice/crio-c306ebfa16f5b2ac0c0183dd991c730f505ed2fe80337abb481296184e4e1501 WatchSource:0}: Error finding container c306ebfa16f5b2ac0c0183dd991c730f505ed2fe80337abb481296184e4e1501: Status 404 returned error can't find the container with id c306ebfa16f5b2ac0c0183dd991c730f505ed2fe80337abb481296184e4e1501
Nov 01 07:16:16.425050 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:16.425016    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9" event={"ID":"9f813809-6f25-43fe-bddd-e0d7ceca00e1","Type":"ContainerStarted","Data":"c306ebfa16f5b2ac0c0183dd991c730f505ed2fe80337abb481296184e4e1501"}
Nov 01 07:16:17.431085 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:17.431002    5510 generic.go:356] "Generic (PLEG): container finished" podID="9f813809-6f25-43fe-bddd-e0d7ceca00e1" containerID="23867d20e5be4fcc92c51f2e784aaa78f42e57b41efedc007a6f681214e06a8e" exitCode=0
Nov 01 07:16:17.431445 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:17.431083    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9" event={"ID":"9f813809-6f25-43fe-bddd-e0d7ceca00e1","Type":"ContainerDied","Data":"23867d20e5be4fcc92c51f2e784aaa78f42e57b41efedc007a6f681214e06a8e"}
Nov 01 07:16:17.544017 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:16:17.543977    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:16:18.569005 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.568982    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:18.662962 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.662930    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-openshift-ca\") pod \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") "
Nov 01 07:16:18.663120 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.663014    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f813809-6f25-43fe-bddd-e0d7ceca00e1-kube-api-access-h6w6t\" (UniqueName: \"kubernetes.io/projected/9f813809-6f25-43fe-bddd-e0d7ceca00e1-kube-api-access-h6w6t\") pod \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") "
Nov 01 07:16:18.663120 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.663070    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-xks-ca\") pod \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\" (UID: \"9f813809-6f25-43fe-bddd-e0d7ceca00e1\") "
Nov 01 07:16:18.663295 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.663271    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-openshift-ca" pod "9f813809-6f25-43fe-bddd-e0d7ceca00e1" (UID: "9f813809-6f25-43fe-bddd-e0d7ceca00e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:16:18.663418 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.663402    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:18.665209 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.665186    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f813809-6f25-43fe-bddd-e0d7ceca00e1-kube-api-access-h6w6t" pod "9f813809-6f25-43fe-bddd-e0d7ceca00e1" (UID: "9f813809-6f25-43fe-bddd-e0d7ceca00e1"). InnerVolumeSpecName "kube-api-access-h6w6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:16:18.665297 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.665233    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-xks-ca" pod "9f813809-6f25-43fe-bddd-e0d7ceca00e1" (UID: "9f813809-6f25-43fe-bddd-e0d7ceca00e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:16:18.764042 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.764014    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6w6t\" (UniqueName: \"kubernetes.io/projected/9f813809-6f25-43fe-bddd-e0d7ceca00e1-kube-api-access-h6w6t\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:18.764042 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:18.764038    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f813809-6f25-43fe-bddd-e0d7ceca00e1-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:19.442190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:19.442153    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9" event={"ID":"9f813809-6f25-43fe-bddd-e0d7ceca00e1","Type":"ContainerDied","Data":"c306ebfa16f5b2ac0c0183dd991c730f505ed2fe80337abb481296184e4e1501"}
Nov 01 07:16:19.442374 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:19.442194    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"
Nov 01 07:16:19.442374 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:19.442207    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c306ebfa16f5b2ac0c0183dd991c730f505ed2fe80337abb481296184e4e1501"
Nov 01 07:16:22.046851 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:22.046790    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"]
Nov 01 07:16:22.095821 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:22.095784    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gv624"]
Nov 01 07:16:23.234427 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:23.234399    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe650a0-e8d5-4b95-844f-cef0158dbc1c" path="/var/lib/kubelet/pods/ebe650a0-e8d5-4b95-844f-cef0158dbc1c/volumes"
Nov 01 07:16:26.289871 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.289839    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"]
Nov 01 07:16:26.290275 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.290266    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9f813809-6f25-43fe-bddd-e0d7ceca00e1" containerName="revoke-keys"
Nov 01 07:16:26.290324 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.290277    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f813809-6f25-43fe-bddd-e0d7ceca00e1" containerName="revoke-keys"
Nov 01 07:16:26.290363 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.290357    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f813809-6f25-43fe-bddd-e0d7ceca00e1" containerName="revoke-keys"
Nov 01 07:16:26.293469 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.293453    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.334164 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.334129    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.334164 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.334166    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.334376 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.334181    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfkwq\" (UniqueName: \"kubernetes.io/projected/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-kube-api-access-zfkwq\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.345498 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.345474    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:16:26.395083 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.395042    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:16:26.395273 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.395080    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:16:26.395273 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.395120    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:16:26.396872 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.396847    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"]
Nov 01 07:16:26.434909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.434866    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.435078 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.434919    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.435078 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.434955    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfkwq\" (UniqueName: \"kubernetes.io/projected/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-kube-api-access-zfkwq\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.435564 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.435545    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.437421 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.437403    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.495681 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.495655    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfkwq\" (UniqueName: \"kubernetes.io/projected/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-kube-api-access-zfkwq\") pod \"maas-api-revoke-sub-d3fba056-8c9jg\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.605512 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.605424    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:26.977504 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:26.977481    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"]
Nov 01 07:16:26.979565 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:16:26.979536    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a5eeea_28d7_49fd_bf2f_48a3259cfd67.slice/crio-7467c9a6697da89896fab36960bb9d318691eea143368247690148d602d3fbb9 WatchSource:0}: Error finding container 7467c9a6697da89896fab36960bb9d318691eea143368247690148d602d3fbb9: Status 404 returned error can't find the container with id 7467c9a6697da89896fab36960bb9d318691eea143368247690148d602d3fbb9
Nov 01 07:16:27.485489 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:27.485455    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg" event={"ID":"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67","Type":"ContainerStarted","Data":"bc77658f24005abe220da51240682bbd883b52c554391797cc580f613445b269"}
Nov 01 07:16:27.485953 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:27.485521    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg" event={"ID":"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67","Type":"ContainerStarted","Data":"7467c9a6697da89896fab36960bb9d318691eea143368247690148d602d3fbb9"}
Nov 01 07:16:27.601408 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:27.601304    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg" podStartSLOduration=1.601289382 podStartE2EDuration="1.601289382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:16:27.599961585 +0000 UTC m=+5424.477934996" watchObservedRunningTime="2026-11-01 07:16:27.601289382 +0000 UTC m=+5424.479262792"
Nov 01 07:16:28.387812 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:28.387776    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:16:28.492288 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:28.492250    5510 generic.go:356] "Generic (PLEG): container finished" podID="c9a5eeea-28d7-49fd-bf2f-48a3259cfd67" containerID="bc77658f24005abe220da51240682bbd883b52c554391797cc580f613445b269" exitCode=0
Nov 01 07:16:28.492685 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:28.492329    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg" event={"ID":"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67","Type":"ContainerDied","Data":"bc77658f24005abe220da51240682bbd883b52c554391797cc580f613445b269"}
Nov 01 07:16:29.960813 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:29.960500    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:30.152909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.150770    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-xks-ca\") pod \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") "
Nov 01 07:16:30.153540 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.153519    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-kube-api-access-zfkwq\" (UniqueName: \"kubernetes.io/projected/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-kube-api-access-zfkwq\") pod \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") "
Nov 01 07:16:30.159031 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.159007    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-openshift-ca\") pod \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\" (UID: \"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67\") "
Nov 01 07:16:30.164042 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.164017    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-openshift-ca" pod "c9a5eeea-28d7-49fd-bf2f-48a3259cfd67" (UID: "c9a5eeea-28d7-49fd-bf2f-48a3259cfd67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:16:30.275854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.251002    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-kube-api-access-zfkwq" pod "c9a5eeea-28d7-49fd-bf2f-48a3259cfd67" (UID: "c9a5eeea-28d7-49fd-bf2f-48a3259cfd67"). InnerVolumeSpecName "kube-api-access-zfkwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:16:30.280856 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.276160    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-xks-ca" pod "c9a5eeea-28d7-49fd-bf2f-48a3259cfd67" (UID: "c9a5eeea-28d7-49fd-bf2f-48a3259cfd67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:16:30.300204 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.300150    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:30.300479 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.300457    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:30.300630 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.300614    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfkwq\" (UniqueName: \"kubernetes.io/projected/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67-kube-api-access-zfkwq\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:30.539168 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.539046    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg" event={"ID":"c9a5eeea-28d7-49fd-bf2f-48a3259cfd67","Type":"ContainerDied","Data":"7467c9a6697da89896fab36960bb9d318691eea143368247690148d602d3fbb9"}
Nov 01 07:16:30.539168 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.539125    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7467c9a6697da89896fab36960bb9d318691eea143368247690148d602d3fbb9"
Nov 01 07:16:30.539637 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:30.539537    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"
Nov 01 07:16:39.087991 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:39.087955    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:16:46.046386 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:46.046350    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"]
Nov 01 07:16:46.096208 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:46.096166    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09cd217-qc2zl"]
Nov 01 07:16:47.234456 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:47.234422    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b93737-4a6e-408c-a51a-de76929ffb90" path="/var/lib/kubelet/pods/e8b93737-4a6e-408c-a51a-de76929ffb90/volumes"
Nov 01 07:16:47.544053 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:16:47.543970    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:16:54.166993 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:16:54.166959    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:17:04.810731 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.810699    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"]
Nov 01 07:17:04.811264 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.811167    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c9a5eeea-28d7-49fd-bf2f-48a3259cfd67" containerName="revoke-keys"
Nov 01 07:17:04.811264 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.811179    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a5eeea-28d7-49fd-bf2f-48a3259cfd67" containerName="revoke-keys"
Nov 01 07:17:04.811264 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.811246    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a5eeea-28d7-49fd-bf2f-48a3259cfd67" containerName="revoke-keys"
Nov 01 07:17:04.814527 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.814510    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:04.866694 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.866660    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:17:04.916082 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.916047    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:17:04.916082 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.916066    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:17:04.916347 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.916094    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:17:04.917822 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.917799    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"]
Nov 01 07:17:04.959784 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.959752    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:04.959956 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.959798    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:04.960011 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:04.959974    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bg5t\" (UniqueName: \"kubernetes.io/projected/76bc1ef9-082f-42e7-ac60-accf4a9b1982-kube-api-access-6bg5t\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:05.061527 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.061437    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bg5t\" (UniqueName: \"kubernetes.io/projected/76bc1ef9-082f-42e7-ac60-accf4a9b1982-kube-api-access-6bg5t\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:05.061527 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.061508    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:05.061773 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.061549    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:05.062283 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.062257    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:05.064084 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.064064    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:05.122215 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.122190    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bg5t\" (UniqueName: \"kubernetes.io/projected/76bc1ef9-082f-42e7-ac60-accf4a9b1982-kube-api-access-6bg5t\") pod \"maas-api-revoke-sub-ae594979-fklx6\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:05.127022 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.126997    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:05.298576 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.298548    5510 scope.go:122] "RemoveContainer" containerID="1a99a340bc0d99bd3e44fa7cdcb2f8666292952bfb8b3dbbdb56bc0bae47f295"
Nov 01 07:17:05.498733 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.498647    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"]
Nov 01 07:17:05.501188 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:17:05.501156    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bc1ef9_082f_42e7_ac60_accf4a9b1982.slice/crio-0e1f210ebcaa49bc31f00fc5aa22cee5d459e47e263c2bb63ba0b274cb032003 WatchSource:0}: Error finding container 0e1f210ebcaa49bc31f00fc5aa22cee5d459e47e263c2bb63ba0b274cb032003: Status 404 returned error can't find the container with id 0e1f210ebcaa49bc31f00fc5aa22cee5d459e47e263c2bb63ba0b274cb032003
Nov 01 07:17:05.769564 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:05.769531    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6" event={"ID":"76bc1ef9-082f-42e7-ac60-accf4a9b1982","Type":"ContainerStarted","Data":"0e1f210ebcaa49bc31f00fc5aa22cee5d459e47e263c2bb63ba0b274cb032003"}
Nov 01 07:17:06.775368 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:06.775333    5510 generic.go:356] "Generic (PLEG): container finished" podID="76bc1ef9-082f-42e7-ac60-accf4a9b1982" containerID="ba63f587e14dad06bf2d0687348dc745954ccb49bd8e11d1cb6e2c4c061e9726" exitCode=0
Nov 01 07:17:06.775753 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:06.775399    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6" event={"ID":"76bc1ef9-082f-42e7-ac60-accf4a9b1982","Type":"ContainerDied","Data":"ba63f587e14dad06bf2d0687348dc745954ccb49bd8e11d1cb6e2c4c061e9726"}
Nov 01 07:17:07.918621 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:07.918593    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:07.989734 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:07.989706    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc1ef9-082f-42e7-ac60-accf4a9b1982-kube-api-access-6bg5t\" (UniqueName: \"kubernetes.io/projected/76bc1ef9-082f-42e7-ac60-accf4a9b1982-kube-api-access-6bg5t\") pod \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") "
Nov 01 07:17:07.989921 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:07.989784    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-xks-ca\") pod \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") "
Nov 01 07:17:07.989921 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:07.989867    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-openshift-ca\") pod \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\" (UID: \"76bc1ef9-082f-42e7-ac60-accf4a9b1982\") "
Nov 01 07:17:07.990239 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:07.990217    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-openshift-ca" pod "76bc1ef9-082f-42e7-ac60-accf4a9b1982" (UID: "76bc1ef9-082f-42e7-ac60-accf4a9b1982"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:17:07.992085 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:07.992064    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc1ef9-082f-42e7-ac60-accf4a9b1982-kube-api-access-6bg5t" pod "76bc1ef9-082f-42e7-ac60-accf4a9b1982" (UID: "76bc1ef9-082f-42e7-ac60-accf4a9b1982"). InnerVolumeSpecName "kube-api-access-6bg5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:17:07.992156 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:07.992118    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-xks-ca" pod "76bc1ef9-082f-42e7-ac60-accf4a9b1982" (UID: "76bc1ef9-082f-42e7-ac60-accf4a9b1982"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:17:08.090686 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:08.090593    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:08.090686 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:08.090633    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bg5t\" (UniqueName: \"kubernetes.io/projected/76bc1ef9-082f-42e7-ac60-accf4a9b1982-kube-api-access-6bg5t\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:08.090686 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:08.090643    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bc1ef9-082f-42e7-ac60-accf4a9b1982-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:08.786608 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:08.786574    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"
Nov 01 07:17:08.786804 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:08.786608    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6" event={"ID":"76bc1ef9-082f-42e7-ac60-accf4a9b1982","Type":"ContainerDied","Data":"0e1f210ebcaa49bc31f00fc5aa22cee5d459e47e263c2bb63ba0b274cb032003"}
Nov 01 07:17:08.786804 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:08.786691    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e1f210ebcaa49bc31f00fc5aa22cee5d459e47e263c2bb63ba0b274cb032003"
Nov 01 07:17:17.543985 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:17:17.543948    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:17:17.674510 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:17.674480    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:17:29.795152 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:29.795121    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:17:33.708983 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.708947    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"]
Nov 01 07:17:33.709391 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.709377    5510 cpu_manager.go:414] "containerMap: removing container" podUID="76bc1ef9-082f-42e7-ac60-accf4a9b1982" containerName="revoke-keys"
Nov 01 07:17:33.709438 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.709392    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bc1ef9-082f-42e7-ac60-accf4a9b1982" containerName="revoke-keys"
Nov 01 07:17:33.709491 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.709481    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bc1ef9-082f-42e7-ac60-accf4a9b1982" containerName="revoke-keys"
Nov 01 07:17:33.712691 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.712677    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.764969 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.764938    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:17:33.814439 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.814407    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.814623 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.814448    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9znmb\" (UniqueName: \"kubernetes.io/projected/b2fba914-6237-489a-8741-0c395365cfa7-kube-api-access-9znmb\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.814623 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.814469    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.814623 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.814506    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:17:33.814623 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.814528    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:17:33.814623 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.814564    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:17:33.816394 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.816371    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"]
Nov 01 07:17:33.915257 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.915225    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9znmb\" (UniqueName: \"kubernetes.io/projected/b2fba914-6237-489a-8741-0c395365cfa7-kube-api-access-9znmb\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.915257 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.915261    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.915464 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.915354    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.916135 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.916115    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.917849 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.917815    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:33.976412 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:33.976348    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9znmb\" (UniqueName: \"kubernetes.io/projected/b2fba914-6237-489a-8741-0c395365cfa7-kube-api-access-9znmb\") pod \"maas-api-revoke-sub-a4aef7d7-pzkz8\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:34.030447 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:34.030424    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:34.403204 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:34.403184    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"]
Nov 01 07:17:34.405841 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:17:34.405800    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fba914_6237_489a_8741_0c395365cfa7.slice/crio-d1cef610ba6475d02027abf8b2c12d02d093e02075a80e3358889597dbf18ff8 WatchSource:0}: Error finding container d1cef610ba6475d02027abf8b2c12d02d093e02075a80e3358889597dbf18ff8: Status 404 returned error can't find the container with id d1cef610ba6475d02027abf8b2c12d02d093e02075a80e3358889597dbf18ff8
Nov 01 07:17:34.948911 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:34.948804    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8" event={"ID":"b2fba914-6237-489a-8741-0c395365cfa7","Type":"ContainerStarted","Data":"23c7a03636c150a10601183b37cc0c517a90a43959b07eb8da897d452e4d78d5"}
Nov 01 07:17:34.948911 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:34.948879    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8" event={"ID":"b2fba914-6237-489a-8741-0c395365cfa7","Type":"ContainerStarted","Data":"d1cef610ba6475d02027abf8b2c12d02d093e02075a80e3358889597dbf18ff8"}
Nov 01 07:17:35.066543 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:35.066490    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8" podStartSLOduration=2.066474809 podStartE2EDuration="2.066474809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:17:35.065304041 +0000 UTC m=+5491.943277452" watchObservedRunningTime="2026-11-01 07:17:35.066474809 +0000 UTC m=+5491.944448219"
Nov 01 07:17:35.955161 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:35.955128    5510 generic.go:356] "Generic (PLEG): container finished" podID="b2fba914-6237-489a-8741-0c395365cfa7" containerID="23c7a03636c150a10601183b37cc0c517a90a43959b07eb8da897d452e4d78d5" exitCode=0
Nov 01 07:17:35.955653 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:35.955192    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8" event={"ID":"b2fba914-6237-489a-8741-0c395365cfa7","Type":"ContainerDied","Data":"23c7a03636c150a10601183b37cc0c517a90a43959b07eb8da897d452e4d78d5"}
Nov 01 07:17:37.102133 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.102106    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:37.248943 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.248856    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-openshift-ca\") pod \"b2fba914-6237-489a-8741-0c395365cfa7\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") "
Nov 01 07:17:37.249109 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.249004    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-xks-ca\") pod \"b2fba914-6237-489a-8741-0c395365cfa7\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") "
Nov 01 07:17:37.249109 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.249035    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fba914-6237-489a-8741-0c395365cfa7-kube-api-access-9znmb\" (UniqueName: \"kubernetes.io/projected/b2fba914-6237-489a-8741-0c395365cfa7-kube-api-access-9znmb\") pod \"b2fba914-6237-489a-8741-0c395365cfa7\" (UID: \"b2fba914-6237-489a-8741-0c395365cfa7\") "
Nov 01 07:17:37.249243 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.249197    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-openshift-ca" pod "b2fba914-6237-489a-8741-0c395365cfa7" (UID: "b2fba914-6237-489a-8741-0c395365cfa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:17:37.249329 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.249302    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:37.251322 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.251293    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fba914-6237-489a-8741-0c395365cfa7-kube-api-access-9znmb" pod "b2fba914-6237-489a-8741-0c395365cfa7" (UID: "b2fba914-6237-489a-8741-0c395365cfa7"). InnerVolumeSpecName "kube-api-access-9znmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:17:37.251322 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.251311    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-xks-ca" pod "b2fba914-6237-489a-8741-0c395365cfa7" (UID: "b2fba914-6237-489a-8741-0c395365cfa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:17:37.350051 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.350020    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fba914-6237-489a-8741-0c395365cfa7-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:37.350051 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.350052    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9znmb\" (UniqueName: \"kubernetes.io/projected/b2fba914-6237-489a-8741-0c395365cfa7-kube-api-access-9znmb\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:37.966099 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.966070    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"
Nov 01 07:17:37.966274 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.966072    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8" event={"ID":"b2fba914-6237-489a-8741-0c395365cfa7","Type":"ContainerDied","Data":"d1cef610ba6475d02027abf8b2c12d02d093e02075a80e3358889597dbf18ff8"}
Nov 01 07:17:37.966274 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:37.966194    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1cef610ba6475d02027abf8b2c12d02d093e02075a80e3358889597dbf18ff8"
Nov 01 07:17:39.073330 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:39.073297    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:17:47.545225 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:17:47.545134    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:17:52.004277 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:52.004242    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:17:54.884853 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:54.884740    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"]
Nov 01 07:17:54.888865 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:54.885406    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b2fba914-6237-489a-8741-0c395365cfa7" containerName="revoke-keys"
Nov 01 07:17:54.888865 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:54.885430    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2fba914-6237-489a-8741-0c395365cfa7" containerName="revoke-keys"
Nov 01 07:17:54.888865 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:54.885549    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2fba914-6237-489a-8741-0c395365cfa7" containerName="revoke-keys"
Nov 01 07:17:54.894100 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:54.894076    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:54.963872 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:54.931589    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"]
Nov 01 07:17:55.042544 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.041363    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:17:55.042544 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.041440    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:17:55.042544 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.041525    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:17:55.042544 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.041620    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:17:55.042544 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.042154    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.042544 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.042238    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.042544 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.042272    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5w4\" (UniqueName: \"kubernetes.io/projected/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-kube-api-access-5r5w4\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.162909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.157576    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.162909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.157693    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.162909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.157727    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5w4\" (UniqueName: \"kubernetes.io/projected/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-kube-api-access-5r5w4\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.164196 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.163718    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.173462 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.173437    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.228284 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.228238    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5w4\" (UniqueName: \"kubernetes.io/projected/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-kube-api-access-5r5w4\") pod \"maas-api-revoke-sub-79805ca9-ns7fh\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.344998 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.344956    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:55.732820 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:55.732633    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"]
Nov 01 07:17:55.735341 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:17:55.735310    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0934cf6_3be6_4d53_ae20_ff21e28ce25f.slice/crio-4e6c379cb6a6743876fcddf6fa65ace707ca013813cbbd9741955af5d941a398 WatchSource:0}: Error finding container 4e6c379cb6a6743876fcddf6fa65ace707ca013813cbbd9741955af5d941a398: Status 404 returned error can't find the container with id 4e6c379cb6a6743876fcddf6fa65ace707ca013813cbbd9741955af5d941a398
Nov 01 07:17:56.094977 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:56.094944    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh" event={"ID":"e0934cf6-3be6-4d53-ae20-ff21e28ce25f","Type":"ContainerStarted","Data":"53f1f500373538dd0a358dbf6777613262dcb958e6c53d077c7f3e80b38bee3d"}
Nov 01 07:17:56.095366 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:56.094998    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh" event={"ID":"e0934cf6-3be6-4d53-ae20-ff21e28ce25f","Type":"ContainerStarted","Data":"4e6c379cb6a6743876fcddf6fa65ace707ca013813cbbd9741955af5d941a398"}
Nov 01 07:17:56.210500 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:56.210443    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh" podStartSLOduration=2.210430123 podStartE2EDuration="2.210430123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:17:56.209177166 +0000 UTC m=+5513.087150575" watchObservedRunningTime="2026-11-01 07:17:56.210430123 +0000 UTC m=+5513.088403533"
Nov 01 07:17:57.101331 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:57.101301    5510 generic.go:356] "Generic (PLEG): container finished" podID="e0934cf6-3be6-4d53-ae20-ff21e28ce25f" containerID="53f1f500373538dd0a358dbf6777613262dcb958e6c53d077c7f3e80b38bee3d" exitCode=0
Nov 01 07:17:57.101783 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:57.101391    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh" event={"ID":"e0934cf6-3be6-4d53-ae20-ff21e28ce25f","Type":"ContainerDied","Data":"53f1f500373538dd0a358dbf6777613262dcb958e6c53d077c7f3e80b38bee3d"}
Nov 01 07:17:58.238889 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.238861    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:58.287886 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.287850    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-kube-api-access-5r5w4\" (UniqueName: \"kubernetes.io/projected/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-kube-api-access-5r5w4\") pod \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") "
Nov 01 07:17:58.288044 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.287894    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-openshift-ca\") pod \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") "
Nov 01 07:17:58.288348 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.288319    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-openshift-ca" pod "e0934cf6-3be6-4d53-ae20-ff21e28ce25f" (UID: "e0934cf6-3be6-4d53-ae20-ff21e28ce25f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:17:58.290278 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.290258    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-kube-api-access-5r5w4" pod "e0934cf6-3be6-4d53-ae20-ff21e28ce25f" (UID: "e0934cf6-3be6-4d53-ae20-ff21e28ce25f"). InnerVolumeSpecName "kube-api-access-5r5w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:17:58.388878 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.388776    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-xks-ca\") pod \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\" (UID: \"e0934cf6-3be6-4d53-ae20-ff21e28ce25f\") "
Nov 01 07:17:58.389094 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.389077    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r5w4\" (UniqueName: \"kubernetes.io/projected/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-kube-api-access-5r5w4\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:58.389182 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.389096    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:58.391085 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.391059    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-xks-ca" pod "e0934cf6-3be6-4d53-ae20-ff21e28ce25f" (UID: "e0934cf6-3be6-4d53-ae20-ff21e28ce25f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:17:58.489971 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:58.489934    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0934cf6-3be6-4d53-ae20-ff21e28ce25f-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:59.112519 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:59.112491    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"
Nov 01 07:17:59.112696 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:59.112521    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh" event={"ID":"e0934cf6-3be6-4d53-ae20-ff21e28ce25f","Type":"ContainerDied","Data":"4e6c379cb6a6743876fcddf6fa65ace707ca013813cbbd9741955af5d941a398"}
Nov 01 07:17:59.112696 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:17:59.112574    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6c379cb6a6743876fcddf6fa65ace707ca013813cbbd9741955af5d941a398"
Nov 01 07:18:01.402276 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:01.402231    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:18:05.319114 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:05.319086    5510 scope.go:122] "RemoveContainer" containerID="b821715021afc401a0761aa1b0f5771c3dfb36a9bf9f9a798a30f9364a8151fa"
Nov 01 07:18:05.327910 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:05.327889    5510 scope.go:122] "RemoveContainer" containerID="30ed291172ba835f2565f66eea6c62d39c954424b4f49895deb711a6dfe05f07"
Nov 01 07:18:10.461787 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:10.461751    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:18:12.393852 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.393418    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"]
Nov 01 07:18:12.397849 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.397419    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e0934cf6-3be6-4d53-ae20-ff21e28ce25f" containerName="revoke-keys"
Nov 01 07:18:12.397849 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.397443    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0934cf6-3be6-4d53-ae20-ff21e28ce25f" containerName="revoke-keys"
Nov 01 07:18:12.397849 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.397583    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0934cf6-3be6-4d53-ae20-ff21e28ce25f" containerName="revoke-keys"
Nov 01 07:18:12.420667 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.420645    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.450188 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.450152    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"]
Nov 01 07:18:12.498436 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.498148    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:18:12.498657 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.498391    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:18:12.506899 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.503069    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:18:12.527000 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.526966    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:18:12.565077 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.565050    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.565306 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.565285    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhzr\" (UniqueName: \"kubernetes.io/projected/e247f873-438f-4f55-a39e-de3fdf398fc5-kube-api-access-qkhzr\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.565521 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.565499    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.682903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.668416    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.682903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.668470    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhzr\" (UniqueName: \"kubernetes.io/projected/e247f873-438f-4f55-a39e-de3fdf398fc5-kube-api-access-qkhzr\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.682903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.668551    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.682903 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.669405    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.710546 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.710511    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.800609 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.800574    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhzr\" (UniqueName: \"kubernetes.io/projected/e247f873-438f-4f55-a39e-de3fdf398fc5-kube-api-access-qkhzr\") pod \"maas-api-revoke-sub-0b7e8edb-cgdg2\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:12.800984 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:12.800965    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:13.745693 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:13.745662    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"]
Nov 01 07:18:13.748720 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:18:13.748693    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode247f873_438f_4f55_a39e_de3fdf398fc5.slice/crio-3983f8b929ddb0bede69e013012032843836025662620bd8018ae156575d9198 WatchSource:0}: Error finding container 3983f8b929ddb0bede69e013012032843836025662620bd8018ae156575d9198: Status 404 returned error can't find the container with id 3983f8b929ddb0bede69e013012032843836025662620bd8018ae156575d9198
Nov 01 07:18:14.474658 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:14.474623    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2" event={"ID":"e247f873-438f-4f55-a39e-de3fdf398fc5","Type":"ContainerStarted","Data":"b34aa9b1922d597d7e5916f3a1ad826329d7fb320072629ec472c37fbb2192ce"}
Nov 01 07:18:14.474847 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:14.474700    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2" event={"ID":"e247f873-438f-4f55-a39e-de3fdf398fc5","Type":"ContainerStarted","Data":"3983f8b929ddb0bede69e013012032843836025662620bd8018ae156575d9198"}
Nov 01 07:18:14.590575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:14.590527    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2" podStartSLOduration=2.590513346 podStartE2EDuration="2.590513346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:18:14.588988499 +0000 UTC m=+5531.466961910" watchObservedRunningTime="2026-11-01 07:18:14.590513346 +0000 UTC m=+5531.468486758"
Nov 01 07:18:15.480641 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:15.480604    5510 generic.go:356] "Generic (PLEG): container finished" podID="e247f873-438f-4f55-a39e-de3fdf398fc5" containerID="b34aa9b1922d597d7e5916f3a1ad826329d7fb320072629ec472c37fbb2192ce" exitCode=0
Nov 01 07:18:15.481090 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:15.480685    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2" event={"ID":"e247f873-438f-4f55-a39e-de3fdf398fc5","Type":"ContainerDied","Data":"b34aa9b1922d597d7e5916f3a1ad826329d7fb320072629ec472c37fbb2192ce"}
Nov 01 07:18:16.623697 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.623673    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:16.689715 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.689685    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-openshift-ca\") pod \"e247f873-438f-4f55-a39e-de3fdf398fc5\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") "
Nov 01 07:18:16.689936 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.689743    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-xks-ca\") pod \"e247f873-438f-4f55-a39e-de3fdf398fc5\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") "
Nov 01 07:18:16.689936 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.689784    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e247f873-438f-4f55-a39e-de3fdf398fc5-kube-api-access-qkhzr\" (UniqueName: \"kubernetes.io/projected/e247f873-438f-4f55-a39e-de3fdf398fc5-kube-api-access-qkhzr\") pod \"e247f873-438f-4f55-a39e-de3fdf398fc5\" (UID: \"e247f873-438f-4f55-a39e-de3fdf398fc5\") "
Nov 01 07:18:16.690134 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.690113    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-openshift-ca" pod "e247f873-438f-4f55-a39e-de3fdf398fc5" (UID: "e247f873-438f-4f55-a39e-de3fdf398fc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:18:16.691988 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.691964    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-xks-ca" pod "e247f873-438f-4f55-a39e-de3fdf398fc5" (UID: "e247f873-438f-4f55-a39e-de3fdf398fc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:18:16.692084 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.692034    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e247f873-438f-4f55-a39e-de3fdf398fc5-kube-api-access-qkhzr" pod "e247f873-438f-4f55-a39e-de3fdf398fc5" (UID: "e247f873-438f-4f55-a39e-de3fdf398fc5"). InnerVolumeSpecName "kube-api-access-qkhzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:18:16.791742 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.791656    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:18:16.791742 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.791689    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e247f873-438f-4f55-a39e-de3fdf398fc5-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:18:16.791742 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:16.791700    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkhzr\" (UniqueName: \"kubernetes.io/projected/e247f873-438f-4f55-a39e-de3fdf398fc5-kube-api-access-qkhzr\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:18:17.491713 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:17.491635    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"
Nov 01 07:18:17.491713 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:17.491674    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2" event={"ID":"e247f873-438f-4f55-a39e-de3fdf398fc5","Type":"ContainerDied","Data":"3983f8b929ddb0bede69e013012032843836025662620bd8018ae156575d9198"}
Nov 01 07:18:17.491940 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:17.491717    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3983f8b929ddb0bede69e013012032843836025662620bd8018ae156575d9198"
Nov 01 07:18:17.544972 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:18:17.544936    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:18:19.709563 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:19.709524    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:18:23.680349 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.680311    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"]
Nov 01 07:18:23.680764 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.680709    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e247f873-438f-4f55-a39e-de3fdf398fc5" containerName="revoke-keys"
Nov 01 07:18:23.680764 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.680720    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e247f873-438f-4f55-a39e-de3fdf398fc5" containerName="revoke-keys"
Nov 01 07:18:23.680916 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.680813    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e247f873-438f-4f55-a39e-de3fdf398fc5" containerName="revoke-keys"
Nov 01 07:18:23.684359 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.684334    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.736470 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.736445    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:18:23.757875 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.757843    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.758037 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.757924    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.758037 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.758025    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrkm\" (UniqueName: \"kubernetes.io/projected/b483d16d-5aa8-479a-8c96-49538a8ab972-kube-api-access-rhrkm\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.786041 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.786008    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:18:23.786265 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.786249    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:18:23.786342 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.786267    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:18:23.788208 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.788191    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"]
Nov 01 07:18:23.858974 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.858945    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.859103 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.859011    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrkm\" (UniqueName: \"kubernetes.io/projected/b483d16d-5aa8-479a-8c96-49538a8ab972-kube-api-access-rhrkm\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.859103 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.859073    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.859728 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.859652    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.861412 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.861394    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.919734 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.919710    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrkm\" (UniqueName: \"kubernetes.io/projected/b483d16d-5aa8-479a-8c96-49538a8ab972-kube-api-access-rhrkm\") pod \"maas-api-revoke-sub-3d4156f1-hclj5\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:23.997675 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:23.997652    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:24.368731 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:24.368704    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"]
Nov 01 07:18:24.371002 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:18:24.370959    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb483d16d_5aa8_479a_8c96_49538a8ab972.slice/crio-2b453a3928ab65d4d50ed139d2e45c5e8a286d27f9ba326f07ca6b6ba10e53f3 WatchSource:0}: Error finding container 2b453a3928ab65d4d50ed139d2e45c5e8a286d27f9ba326f07ca6b6ba10e53f3: Status 404 returned error can't find the container with id 2b453a3928ab65d4d50ed139d2e45c5e8a286d27f9ba326f07ca6b6ba10e53f3
Nov 01 07:18:24.525411 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:24.525381    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5" event={"ID":"b483d16d-5aa8-479a-8c96-49538a8ab972","Type":"ContainerStarted","Data":"2b453a3928ab65d4d50ed139d2e45c5e8a286d27f9ba326f07ca6b6ba10e53f3"}
Nov 01 07:18:25.532182 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:25.532143    5510 generic.go:356] "Generic (PLEG): container finished" podID="b483d16d-5aa8-479a-8c96-49538a8ab972" containerID="0c9bf35b5288ed20876be908410a03fb5220278435b1a8b3a6dab097af15b77c" exitCode=0
Nov 01 07:18:25.532607 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:25.532228    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5" event={"ID":"b483d16d-5aa8-479a-8c96-49538a8ab972","Type":"ContainerDied","Data":"0c9bf35b5288ed20876be908410a03fb5220278435b1a8b3a6dab097af15b77c"}
Nov 01 07:18:26.675806 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.675784    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:26.789433 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.789369    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b483d16d-5aa8-479a-8c96-49538a8ab972-kube-api-access-rhrkm\" (UniqueName: \"kubernetes.io/projected/b483d16d-5aa8-479a-8c96-49538a8ab972-kube-api-access-rhrkm\") pod \"b483d16d-5aa8-479a-8c96-49538a8ab972\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") "
Nov 01 07:18:26.789617 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.789450    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-xks-ca\") pod \"b483d16d-5aa8-479a-8c96-49538a8ab972\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") "
Nov 01 07:18:26.789617 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.789524    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-openshift-ca\") pod \"b483d16d-5aa8-479a-8c96-49538a8ab972\" (UID: \"b483d16d-5aa8-479a-8c96-49538a8ab972\") "
Nov 01 07:18:26.789915 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.789888    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-openshift-ca" pod "b483d16d-5aa8-479a-8c96-49538a8ab972" (UID: "b483d16d-5aa8-479a-8c96-49538a8ab972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:18:26.792176 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.792147    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-xks-ca" pod "b483d16d-5aa8-479a-8c96-49538a8ab972" (UID: "b483d16d-5aa8-479a-8c96-49538a8ab972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:18:26.792708 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.792685    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b483d16d-5aa8-479a-8c96-49538a8ab972-kube-api-access-rhrkm" pod "b483d16d-5aa8-479a-8c96-49538a8ab972" (UID: "b483d16d-5aa8-479a-8c96-49538a8ab972"). InnerVolumeSpecName "kube-api-access-rhrkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:18:26.891014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.890985    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrkm\" (UniqueName: \"kubernetes.io/projected/b483d16d-5aa8-479a-8c96-49538a8ab972-kube-api-access-rhrkm\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:18:26.891014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.891009    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:18:26.891182 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:26.891021    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b483d16d-5aa8-479a-8c96-49538a8ab972-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:18:27.543546 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:27.543458    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"
Nov 01 07:18:27.543546 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:27.543505    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5" event={"ID":"b483d16d-5aa8-479a-8c96-49538a8ab972","Type":"ContainerDied","Data":"2b453a3928ab65d4d50ed139d2e45c5e8a286d27f9ba326f07ca6b6ba10e53f3"}
Nov 01 07:18:27.543795 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:27.543556    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b453a3928ab65d4d50ed139d2e45c5e8a286d27f9ba326f07ca6b6ba10e53f3"
Nov 01 07:18:36.499394 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:36.499355    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:18:47.544916 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:18:47.544871    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:18:47.717441 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:18:47.717413    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:19:01.679259 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.679186    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"]
Nov 01 07:19:01.728138 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.728107    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hclj5"]
Nov 01 07:19:01.799592 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.799555    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"]
Nov 01 07:19:01.800048 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.800025    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b483d16d-5aa8-479a-8c96-49538a8ab972" containerName="revoke-keys"
Nov 01 07:19:01.800048 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.800048    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b483d16d-5aa8-479a-8c96-49538a8ab972" containerName="revoke-keys"
Nov 01 07:19:01.800213 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.800176    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b483d16d-5aa8-479a-8c96-49538a8ab972" containerName="revoke-keys"
Nov 01 07:19:01.803366 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.803352    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:01.855584 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.855549    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:19:01.855766 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.855678    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:19:01.855766 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.855677    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:19:01.855766 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.855720    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:19:01.861442 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.861403    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"]
Nov 01 07:19:01.913867 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.913808    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:01.914038 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.913972    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmbf\" (UniqueName: \"kubernetes.io/projected/35ed87ad-dd79-4f39-9bc4-f1a449efc021-kube-api-access-pbmbf\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:01.914082 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:01.914042    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:02.014715 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.014678    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmbf\" (UniqueName: \"kubernetes.io/projected/35ed87ad-dd79-4f39-9bc4-f1a449efc021-kube-api-access-pbmbf\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:02.014955 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.014743    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:02.014955 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.014778    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:02.015404 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.015383    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:02.017372 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.017350    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:02.075688 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.075660    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmbf\" (UniqueName: \"kubernetes.io/projected/35ed87ad-dd79-4f39-9bc4-f1a449efc021-kube-api-access-pbmbf\") pod \"maas-api-revoke-sub-3d4156f1-dmn4t\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:02.116055 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.116032    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:02.488005 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.487859    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"]
Nov 01 07:19:02.490349 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:19:02.490319    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ed87ad_dd79_4f39_9bc4_f1a449efc021.slice/crio-27105bbc8b31fea860d18bac411ea5d1811022567639d620067a339a0e62e875 WatchSource:0}: Error finding container 27105bbc8b31fea860d18bac411ea5d1811022567639d620067a339a0e62e875: Status 404 returned error can't find the container with id 27105bbc8b31fea860d18bac411ea5d1811022567639d620067a339a0e62e875
Nov 01 07:19:02.751922 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:02.751819    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t" event={"ID":"35ed87ad-dd79-4f39-9bc4-f1a449efc021","Type":"ContainerStarted","Data":"27105bbc8b31fea860d18bac411ea5d1811022567639d620067a339a0e62e875"}
Nov 01 07:19:03.234166 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:03.234132    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b483d16d-5aa8-479a-8c96-49538a8ab972" path="/var/lib/kubelet/pods/b483d16d-5aa8-479a-8c96-49538a8ab972/volumes"
Nov 01 07:19:03.757695 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:03.757665    5510 generic.go:356] "Generic (PLEG): container finished" podID="35ed87ad-dd79-4f39-9bc4-f1a449efc021" containerID="2ddc2f9c1ea3cbb5ef521b25f1442fab5a9239dd6b653af871b7936b069ed2c0" exitCode=0
Nov 01 07:19:03.758145 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:03.757747    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t" event={"ID":"35ed87ad-dd79-4f39-9bc4-f1a449efc021","Type":"ContainerDied","Data":"2ddc2f9c1ea3cbb5ef521b25f1442fab5a9239dd6b653af871b7936b069ed2c0"}
Nov 01 07:19:04.902090 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:04.902063    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:04.939644 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:04.939615    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ed87ad-dd79-4f39-9bc4-f1a449efc021-kube-api-access-pbmbf\" (UniqueName: \"kubernetes.io/projected/35ed87ad-dd79-4f39-9bc4-f1a449efc021-kube-api-access-pbmbf\") pod \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") "
Nov 01 07:19:04.939771 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:04.939682    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-openshift-ca\") pod \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") "
Nov 01 07:19:04.939869 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:04.939795    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-xks-ca\") pod \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\" (UID: \"35ed87ad-dd79-4f39-9bc4-f1a449efc021\") "
Nov 01 07:19:04.940035 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:04.940015    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-openshift-ca" pod "35ed87ad-dd79-4f39-9bc4-f1a449efc021" (UID: "35ed87ad-dd79-4f39-9bc4-f1a449efc021"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:19:04.940164 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:04.940147    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:04.941816 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:04.941796    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-xks-ca" pod "35ed87ad-dd79-4f39-9bc4-f1a449efc021" (UID: "35ed87ad-dd79-4f39-9bc4-f1a449efc021"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:19:04.941906 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:04.941890    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ed87ad-dd79-4f39-9bc4-f1a449efc021-kube-api-access-pbmbf" pod "35ed87ad-dd79-4f39-9bc4-f1a449efc021" (UID: "35ed87ad-dd79-4f39-9bc4-f1a449efc021"). InnerVolumeSpecName "kube-api-access-pbmbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:19:05.040734 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:05.040659    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35ed87ad-dd79-4f39-9bc4-f1a449efc021-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:05.040734 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:05.040688    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbmbf\" (UniqueName: \"kubernetes.io/projected/35ed87ad-dd79-4f39-9bc4-f1a449efc021-kube-api-access-pbmbf\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:05.769237 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:05.769208    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"
Nov 01 07:19:05.769414 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:05.769248    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t" event={"ID":"35ed87ad-dd79-4f39-9bc4-f1a449efc021","Type":"ContainerDied","Data":"27105bbc8b31fea860d18bac411ea5d1811022567639d620067a339a0e62e875"}
Nov 01 07:19:05.769414 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:05.769297    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27105bbc8b31fea860d18bac411ea5d1811022567639d620067a339a0e62e875"
Nov 01 07:19:17.544940 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:19:17.544859    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:19:27.689141 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:27.689109    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:19:37.047356 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:37.047312    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"]
Nov 01 07:19:37.096216 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:37.096180    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-vpmz6"]
Nov 01 07:19:37.234106 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:37.234077    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7c5e52-80b2-470a-9293-3e267fea972a" path="/var/lib/kubelet/pods/8d7c5e52-80b2-470a-9293-3e267fea972a/volumes"
Nov 01 07:19:43.355695 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.355664    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"]
Nov 01 07:19:43.356208 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.356114    5510 cpu_manager.go:414] "containerMap: removing container" podUID="35ed87ad-dd79-4f39-9bc4-f1a449efc021" containerName="revoke-keys"
Nov 01 07:19:43.356208 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.356127    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ed87ad-dd79-4f39-9bc4-f1a449efc021" containerName="revoke-keys"
Nov 01 07:19:43.356208 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.356199    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ed87ad-dd79-4f39-9bc4-f1a449efc021" containerName="revoke-keys"
Nov 01 07:19:43.359411 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.359388    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.411462 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.411431    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:19:43.431151 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.431112    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6dmt\" (UniqueName: \"kubernetes.io/projected/32577a4e-9529-4489-9312-711ede5fa50e-kube-api-access-j6dmt\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.431295 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.431225    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.431295 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.431278    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.461016 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.460981    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:19:43.461177 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.461099    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:19:43.461177 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.461132    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:19:43.463067 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.463047    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"]
Nov 01 07:19:43.532122 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.532094    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dmt\" (UniqueName: \"kubernetes.io/projected/32577a4e-9529-4489-9312-711ede5fa50e-kube-api-access-j6dmt\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.532317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.532132    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.532317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.532167    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.532856 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.532805    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.534714 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.534694    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.592932 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.592901    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6dmt\" (UniqueName: \"kubernetes.io/projected/32577a4e-9529-4489-9312-711ede5fa50e-kube-api-access-j6dmt\") pod \"maas-api-revoke-sub-e2fc9c5b-b8lsm\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:43.671787 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:43.671719    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:44.043426 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:44.043399    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"]
Nov 01 07:19:44.045303 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:19:44.045271    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32577a4e_9529_4489_9312_711ede5fa50e.slice/crio-1e63f9f169cb75c26894e996c56ceba96ffc1e9a182ed5ce9ea6b404fecc3a8a WatchSource:0}: Error finding container 1e63f9f169cb75c26894e996c56ceba96ffc1e9a182ed5ce9ea6b404fecc3a8a: Status 404 returned error can't find the container with id 1e63f9f169cb75c26894e996c56ceba96ffc1e9a182ed5ce9ea6b404fecc3a8a
Nov 01 07:19:44.987154 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:44.987122    5510 generic.go:356] "Generic (PLEG): container finished" podID="32577a4e-9529-4489-9312-711ede5fa50e" containerID="5fd453cb6130f1d13262b25b30cdb07d5cb9bd6c8a9729bc6e1ee0dffa837657" exitCode=0
Nov 01 07:19:44.987522 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:44.987207    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm" event={"ID":"32577a4e-9529-4489-9312-711ede5fa50e","Type":"ContainerDied","Data":"5fd453cb6130f1d13262b25b30cdb07d5cb9bd6c8a9729bc6e1ee0dffa837657"}
Nov 01 07:19:44.987522 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:44.987260    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm" event={"ID":"32577a4e-9529-4489-9312-711ede5fa50e","Type":"ContainerStarted","Data":"1e63f9f169cb75c26894e996c56ceba96ffc1e9a182ed5ce9ea6b404fecc3a8a"}
Nov 01 07:19:46.127817 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.127795    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:46.263460 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.263375    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32577a4e-9529-4489-9312-711ede5fa50e-kube-api-access-j6dmt\" (UniqueName: \"kubernetes.io/projected/32577a4e-9529-4489-9312-711ede5fa50e-kube-api-access-j6dmt\") pod \"32577a4e-9529-4489-9312-711ede5fa50e\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") "
Nov 01 07:19:46.263460 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.263429    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-xks-ca\") pod \"32577a4e-9529-4489-9312-711ede5fa50e\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") "
Nov 01 07:19:46.263697 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.263557    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-openshift-ca\") pod \"32577a4e-9529-4489-9312-711ede5fa50e\" (UID: \"32577a4e-9529-4489-9312-711ede5fa50e\") "
Nov 01 07:19:46.263937 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.263911    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-openshift-ca" pod "32577a4e-9529-4489-9312-711ede5fa50e" (UID: "32577a4e-9529-4489-9312-711ede5fa50e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:19:46.265715 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.265692    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32577a4e-9529-4489-9312-711ede5fa50e-kube-api-access-j6dmt" pod "32577a4e-9529-4489-9312-711ede5fa50e" (UID: "32577a4e-9529-4489-9312-711ede5fa50e"). InnerVolumeSpecName "kube-api-access-j6dmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:19:46.265814 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.265723    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-xks-ca" pod "32577a4e-9529-4489-9312-711ede5fa50e" (UID: "32577a4e-9529-4489-9312-711ede5fa50e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:19:46.364547 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.364512    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:46.364547 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.364538    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32577a4e-9529-4489-9312-711ede5fa50e-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:46.364547 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.364547    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6dmt\" (UniqueName: \"kubernetes.io/projected/32577a4e-9529-4489-9312-711ede5fa50e-kube-api-access-j6dmt\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:46.999153 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.999116    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm" event={"ID":"32577a4e-9529-4489-9312-711ede5fa50e","Type":"ContainerDied","Data":"1e63f9f169cb75c26894e996c56ceba96ffc1e9a182ed5ce9ea6b404fecc3a8a"}
Nov 01 07:19:46.999153 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.999137    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"
Nov 01 07:19:46.999381 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:46.999189    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e63f9f169cb75c26894e996c56ceba96ffc1e9a182ed5ce9ea6b404fecc3a8a"
Nov 01 07:19:47.544333 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:19:47.544299    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:19:54.154277 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.154237    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"]
Nov 01 07:19:54.155190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.154904    5510 cpu_manager.go:414] "containerMap: removing container" podUID="32577a4e-9529-4489-9312-711ede5fa50e" containerName="revoke-keys"
Nov 01 07:19:54.155190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.154923    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32577a4e-9529-4489-9312-711ede5fa50e" containerName="revoke-keys"
Nov 01 07:19:54.155190 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.155033    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="32577a4e-9529-4489-9312-711ede5fa50e" containerName="revoke-keys"
Nov 01 07:19:54.158294 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.158272    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.210507 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.210461    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:19:54.238465 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.238429    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htckh\" (UniqueName: \"kubernetes.io/projected/a8ec81c2-7cc9-49cd-a827-90204ef95311-kube-api-access-htckh\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.238643 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.238475    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.238643 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.238610    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.260015 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.259980    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:19:54.260171 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.260042    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:19:54.260171 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.260070    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:19:54.262062 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.262042    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"]
Nov 01 07:19:54.339614 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.339582    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.339879 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.339858    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htckh\" (UniqueName: \"kubernetes.io/projected/a8ec81c2-7cc9-49cd-a827-90204ef95311-kube-api-access-htckh\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.339999 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.339946    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.340359 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.340337    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.342364 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.342327    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.400734 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.400702    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htckh\" (UniqueName: \"kubernetes.io/projected/a8ec81c2-7cc9-49cd-a827-90204ef95311-kube-api-access-htckh\") pod \"maas-api-revoke-sub-5a29efb3-m77zv\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.470794 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.470767    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:54.849353 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:54.849311    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"]
Nov 01 07:19:54.850284 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:19:54.850259    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ec81c2_7cc9_49cd_a827_90204ef95311.slice/crio-05782a3f9548b76d5f5ce0a91420d7b6f309c19f1ba6753ab42b868840c5e79f WatchSource:0}: Error finding container 05782a3f9548b76d5f5ce0a91420d7b6f309c19f1ba6753ab42b868840c5e79f: Status 404 returned error can't find the container with id 05782a3f9548b76d5f5ce0a91420d7b6f309c19f1ba6753ab42b868840c5e79f
Nov 01 07:19:55.039843 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:55.039790    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv" event={"ID":"a8ec81c2-7cc9-49cd-a827-90204ef95311","Type":"ContainerStarted","Data":"05782a3f9548b76d5f5ce0a91420d7b6f309c19f1ba6753ab42b868840c5e79f"}
Nov 01 07:19:56.046142 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:56.046106    5510 generic.go:356] "Generic (PLEG): container finished" podID="a8ec81c2-7cc9-49cd-a827-90204ef95311" containerID="aa8d511869574c10e25bf5e4036189f9cc6555a778fdc2c98aab83333f46a56d" exitCode=0
Nov 01 07:19:56.046543 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:56.046188    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv" event={"ID":"a8ec81c2-7cc9-49cd-a827-90204ef95311","Type":"ContainerDied","Data":"aa8d511869574c10e25bf5e4036189f9cc6555a778fdc2c98aab83333f46a56d"}
Nov 01 07:19:56.269813 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:56.269780    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:19:57.254575 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.254534    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:19:57.375699 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.374746    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-openshift-ca\") pod \"a8ec81c2-7cc9-49cd-a827-90204ef95311\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") "
Nov 01 07:19:57.375699 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.374799    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-xks-ca\") pod \"a8ec81c2-7cc9-49cd-a827-90204ef95311\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") "
Nov 01 07:19:57.375699 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.374871    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ec81c2-7cc9-49cd-a827-90204ef95311-kube-api-access-htckh\" (UniqueName: \"kubernetes.io/projected/a8ec81c2-7cc9-49cd-a827-90204ef95311-kube-api-access-htckh\") pod \"a8ec81c2-7cc9-49cd-a827-90204ef95311\" (UID: \"a8ec81c2-7cc9-49cd-a827-90204ef95311\") "
Nov 01 07:19:57.375699 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.375532    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-openshift-ca" pod "a8ec81c2-7cc9-49cd-a827-90204ef95311" (UID: "a8ec81c2-7cc9-49cd-a827-90204ef95311"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:19:57.380256 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.380212    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-xks-ca" pod "a8ec81c2-7cc9-49cd-a827-90204ef95311" (UID: "a8ec81c2-7cc9-49cd-a827-90204ef95311"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:19:57.381523 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.381481    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ec81c2-7cc9-49cd-a827-90204ef95311-kube-api-access-htckh" pod "a8ec81c2-7cc9-49cd-a827-90204ef95311" (UID: "a8ec81c2-7cc9-49cd-a827-90204ef95311"). InnerVolumeSpecName "kube-api-access-htckh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:19:57.492855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.492537    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:57.492855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.492581    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ec81c2-7cc9-49cd-a827-90204ef95311-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:57.492855 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:57.492600    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htckh\" (UniqueName: \"kubernetes.io/projected/a8ec81c2-7cc9-49cd-a827-90204ef95311-kube-api-access-htckh\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:58.073861 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:58.070035    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv" event={"ID":"a8ec81c2-7cc9-49cd-a827-90204ef95311","Type":"ContainerDied","Data":"05782a3f9548b76d5f5ce0a91420d7b6f309c19f1ba6753ab42b868840c5e79f"}
Nov 01 07:19:58.073861 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:58.070101    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05782a3f9548b76d5f5ce0a91420d7b6f309c19f1ba6753ab42b868840c5e79f"
Nov 01 07:19:58.073861 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:19:58.070173    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"
Nov 01 07:20:06.996618 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:06.996581    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:20:17.545366 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:17.545329    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:20:31.049451 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.049416    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"]
Nov 01 07:20:31.098736 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.098703    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zk2qb"]
Nov 01 07:20:31.237019 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.235514    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74ab553-ac6b-49e6-913a-77fa38cc49a1" path="/var/lib/kubelet/pods/d74ab553-ac6b-49e6-913a-77fa38cc49a1/volumes"
Nov 01 07:20:31.292427 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.292392    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"]
Nov 01 07:20:31.297549 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.297521    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dmn4t"]
Nov 01 07:20:31.418015 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.417929    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"]
Nov 01 07:20:31.418334 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.418321    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a8ec81c2-7cc9-49cd-a827-90204ef95311" containerName="revoke-keys"
Nov 01 07:20:31.418386 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.418335    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ec81c2-7cc9-49cd-a827-90204ef95311" containerName="revoke-keys"
Nov 01 07:20:31.418434 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.418418    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ec81c2-7cc9-49cd-a827-90204ef95311" containerName="revoke-keys"
Nov 01 07:20:31.421815 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.421799    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.474813 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.474781    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:20:31.475004 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.474986    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:20:31.475069 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.474985    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:20:31.475172 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.475160    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:20:31.481394 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.481368    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"]
Nov 01 07:20:31.569340 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.569301    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrsr\" (UniqueName: \"kubernetes.io/projected/1386725d-8002-4035-b052-0d037c22d1fe-kube-api-access-7zrsr\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.569534 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.569450    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.569601 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.569573    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.670946 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.670861    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.670946 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.670912    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrsr\" (UniqueName: \"kubernetes.io/projected/1386725d-8002-4035-b052-0d037c22d1fe-kube-api-access-7zrsr\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.671195 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.671016    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.671664 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.671639    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.673363 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.673342    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.732333 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.732293    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrsr\" (UniqueName: \"kubernetes.io/projected/1386725d-8002-4035-b052-0d037c22d1fe-kube-api-access-7zrsr\") pod \"maas-api-revoke-sub-3d4156f1-ckwfb\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:31.734909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:31.734888    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:33.234765 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:33.234720    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ed87ad-dd79-4f39-9bc4-f1a449efc021" path="/var/lib/kubelet/pods/35ed87ad-dd79-4f39-9bc4-f1a449efc021/volumes"
Nov 01 07:20:34.328173 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:34.328140    5510 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 07:20:34.328173 ip-10-0-138-190 kubenswrapper[5510]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra_1386725d-8002-4035-b052-0d037c22d1fe_0(352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc): error adding pod odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-ckwfb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc" Netns:"/var/run/netns/ba64322d-4bc9-4b55-a1a4-2684a58309be" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-sub-3d4156f1-ckwfb;K8S_POD_INFRA_CONTAINER_ID=352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc;K8S_POD_UID=1386725d-8002-4035-b052-0d037c22d1fe" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb/1386725d-8002-4035-b052-0d037c22d1fe]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-sub-3d4156f1-ckwfb?timeout=1m0s": context deadline exceeded
Nov 01 07:20:34.328173 ip-10-0-138-190 kubenswrapper[5510]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 07:20:34.328173 ip-10-0-138-190 kubenswrapper[5510]:  >
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:34.328214    5510 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra_1386725d-8002-4035-b052-0d037c22d1fe_0(352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc): error adding pod odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-ckwfb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc" Netns:"/var/run/netns/ba64322d-4bc9-4b55-a1a4-2684a58309be" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-sub-3d4156f1-ckwfb;K8S_POD_INFRA_CONTAINER_ID=352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc;K8S_POD_UID=1386725d-8002-4035-b052-0d037c22d1fe" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb/1386725d-8002-4035-b052-0d037c22d1fe]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-sub-3d4156f1-ckwfb?timeout=1m0s": context deadline exceeded
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]:  > pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:34.328229    5510 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra_1386725d-8002-4035-b052-0d037c22d1fe_0(352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc): error adding pod odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-ckwfb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc" Netns:"/var/run/netns/ba64322d-4bc9-4b55-a1a4-2684a58309be" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-sub-3d4156f1-ckwfb;K8S_POD_INFRA_CONTAINER_ID=352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc;K8S_POD_UID=1386725d-8002-4035-b052-0d037c22d1fe" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb/1386725d-8002-4035-b052-0d037c22d1fe]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-sub-3d4156f1-ckwfb?timeout=1m0s": context deadline exceeded
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]:  > pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:34.328544 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:34.328316    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra(1386725d-8002-4035-b052-0d037c22d1fe)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra(1386725d-8002-4035-b052-0d037c22d1fe)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra_1386725d-8002-4035-b052-0d037c22d1fe_0(352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc): error adding pod odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-ckwfb to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc\\\" Netns:\\\"/var/run/netns/ba64322d-4bc9-4b55-a1a4-2684a58309be\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-sub-3d4156f1-ckwfb;K8S_POD_INFRA_CONTAINER_ID=352ae5c0ad2067252588c1d50ffbb86be45eb8df18ee93de1c9eb40995e88ddc;K8S_POD_UID=1386725d-8002-4035-b052-0d037c22d1fe\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb/1386725d-8002-4035-b052-0d037c22d1fe]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-sub-3d4156f1-ckwfb?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb" podUID="1386725d-8002-4035-b052-0d037c22d1fe"
Nov 01 07:20:35.273329 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:35.273294    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:35.273615 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:35.273603    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:37.868541 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:37.868507    5510 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 07:20:37.868541 ip-10-0-138-190 kubenswrapper[5510]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra_1386725d-8002-4035-b052-0d037c22d1fe_0(3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5): error adding pod odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-ckwfb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5" Netns:"/var/run/netns/b1141acb-a373-4db7-ac76-3dc79f36ab2a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-sub-3d4156f1-ckwfb;K8S_POD_INFRA_CONTAINER_ID=3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5;K8S_POD_UID=1386725d-8002-4035-b052-0d037c22d1fe" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb/1386725d-8002-4035-b052-0d037c22d1fe]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-sub-3d4156f1-ckwfb?timeout=1m0s": context deadline exceeded
Nov 01 07:20:37.868541 ip-10-0-138-190 kubenswrapper[5510]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 07:20:37.868541 ip-10-0-138-190 kubenswrapper[5510]:  >
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:37.868563    5510 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra_1386725d-8002-4035-b052-0d037c22d1fe_0(3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5): error adding pod odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-ckwfb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5" Netns:"/var/run/netns/b1141acb-a373-4db7-ac76-3dc79f36ab2a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-sub-3d4156f1-ckwfb;K8S_POD_INFRA_CONTAINER_ID=3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5;K8S_POD_UID=1386725d-8002-4035-b052-0d037c22d1fe" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb/1386725d-8002-4035-b052-0d037c22d1fe]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-sub-3d4156f1-ckwfb?timeout=1m0s": context deadline exceeded
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]:  > pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:37.868583    5510 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra_1386725d-8002-4035-b052-0d037c22d1fe_0(3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5): error adding pod odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-ckwfb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5" Netns:"/var/run/netns/b1141acb-a373-4db7-ac76-3dc79f36ab2a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-sub-3d4156f1-ckwfb;K8S_POD_INFRA_CONTAINER_ID=3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5;K8S_POD_UID=1386725d-8002-4035-b052-0d037c22d1fe" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb/1386725d-8002-4035-b052-0d037c22d1fe]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-sub-3d4156f1-ckwfb?timeout=1m0s": context deadline exceeded
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]:  > pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:37.868958 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:37.868647    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra(1386725d-8002-4035-b052-0d037c22d1fe)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra(1386725d-8002-4035-b052-0d037c22d1fe)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-revoke-sub-3d4156f1-ckwfb_odh-ai-gateway-infra_1386725d-8002-4035-b052-0d037c22d1fe_0(3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5): error adding pod odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-ckwfb to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5\\\" Netns:\\\"/var/run/netns/b1141acb-a373-4db7-ac76-3dc79f36ab2a\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-revoke-sub-3d4156f1-ckwfb;K8S_POD_INFRA_CONTAINER_ID=3322d419b7c5fd9d5698476c07a51e8211ec271c00ac1b2ff1e7e949abaad2e5;K8S_POD_UID=1386725d-8002-4035-b052-0d037c22d1fe\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb] networking: Multus: [odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb/1386725d-8002-4035-b052-0d037c22d1fe]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-revoke-sub-3d4156f1-ckwfb?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb" podUID="1386725d-8002-4035-b052-0d037c22d1fe"
Nov 01 07:20:47.544725 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:20:47.544640    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:20:53.231779 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:53.231744    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:53.232248 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:53.232002    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:54.156469 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:54.156447    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"]
Nov 01 07:20:54.158701 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:20:54.158669    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1386725d_8002_4035_b052_0d037c22d1fe.slice/crio-a0201b3aa4c5d8dfe4cb3a30980f56893ce33f558f3441a702b30d8eb5a31972 WatchSource:0}: Error finding container a0201b3aa4c5d8dfe4cb3a30980f56893ce33f558f3441a702b30d8eb5a31972: Status 404 returned error can't find the container with id a0201b3aa4c5d8dfe4cb3a30980f56893ce33f558f3441a702b30d8eb5a31972
Nov 01 07:20:54.374434 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:54.374400    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb" event={"ID":"1386725d-8002-4035-b052-0d037c22d1fe","Type":"ContainerStarted","Data":"a0201b3aa4c5d8dfe4cb3a30980f56893ce33f558f3441a702b30d8eb5a31972"}
Nov 01 07:20:55.381073 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:55.380992    5510 generic.go:356] "Generic (PLEG): container finished" podID="1386725d-8002-4035-b052-0d037c22d1fe" containerID="3da0054e8a643da603e59d7b486e8bddcc9c47e794308ce031825333c0467cf5" exitCode=0
Nov 01 07:20:55.381430 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:55.381072    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb" event={"ID":"1386725d-8002-4035-b052-0d037c22d1fe","Type":"ContainerDied","Data":"3da0054e8a643da603e59d7b486e8bddcc9c47e794308ce031825333c0467cf5"}
Nov 01 07:20:56.516840 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.516805    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:56.615041 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.615011    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-openshift-ca\") pod \"1386725d-8002-4035-b052-0d037c22d1fe\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") "
Nov 01 07:20:56.615232 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.615070    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-xks-ca\") pod \"1386725d-8002-4035-b052-0d037c22d1fe\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") "
Nov 01 07:20:56.615232 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.615165    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1386725d-8002-4035-b052-0d037c22d1fe-kube-api-access-7zrsr\" (UniqueName: \"kubernetes.io/projected/1386725d-8002-4035-b052-0d037c22d1fe-kube-api-access-7zrsr\") pod \"1386725d-8002-4035-b052-0d037c22d1fe\" (UID: \"1386725d-8002-4035-b052-0d037c22d1fe\") "
Nov 01 07:20:56.615382 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.615358    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-openshift-ca" pod "1386725d-8002-4035-b052-0d037c22d1fe" (UID: "1386725d-8002-4035-b052-0d037c22d1fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:20:56.617288 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.617266    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-xks-ca" pod "1386725d-8002-4035-b052-0d037c22d1fe" (UID: "1386725d-8002-4035-b052-0d037c22d1fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:20:56.617506 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.617489    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1386725d-8002-4035-b052-0d037c22d1fe-kube-api-access-7zrsr" pod "1386725d-8002-4035-b052-0d037c22d1fe" (UID: "1386725d-8002-4035-b052-0d037c22d1fe"). InnerVolumeSpecName "kube-api-access-7zrsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:20:56.716718 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.716689    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zrsr\" (UniqueName: \"kubernetes.io/projected/1386725d-8002-4035-b052-0d037c22d1fe-kube-api-access-7zrsr\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:56.716718 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.716719    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:56.716947 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:56.716732    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1386725d-8002-4035-b052-0d037c22d1fe-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:57.396267 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:57.396240    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"
Nov 01 07:20:57.396422 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:57.396279    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb" event={"ID":"1386725d-8002-4035-b052-0d037c22d1fe","Type":"ContainerDied","Data":"a0201b3aa4c5d8dfe4cb3a30980f56893ce33f558f3441a702b30d8eb5a31972"}
Nov 01 07:20:57.396422 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:20:57.396324    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0201b3aa4c5d8dfe4cb3a30980f56893ce33f558f3441a702b30d8eb5a31972"
Nov 01 07:21:04.396381 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:04.396342    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:21:05.379657 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:05.379623    5510 scope.go:122] "RemoveContainer" containerID="a3fb9f5aaaf41f8ebc20c25efa39fb0f6d413d0b850fba20cfb86c18f4e44691"
Nov 01 07:21:15.689502 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:15.689469    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:21:17.544764 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:21:17.544726    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:21:19.052556 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:19.052514    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"]
Nov 01 07:21:19.054328 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:19.054302    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f47c9"]
Nov 01 07:21:19.234367 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:19.234337    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f813809-6f25-43fe-bddd-e0d7ceca00e1" path="/var/lib/kubelet/pods/9f813809-6f25-43fe-bddd-e0d7ceca00e1/volumes"
Nov 01 07:21:23.404843 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.404784    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"]
Nov 01 07:21:23.405414 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.405394    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1386725d-8002-4035-b052-0d037c22d1fe" containerName="revoke-keys"
Nov 01 07:21:23.405500 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.405417    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1386725d-8002-4035-b052-0d037c22d1fe" containerName="revoke-keys"
Nov 01 07:21:23.405558 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.405539    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1386725d-8002-4035-b052-0d037c22d1fe" containerName="revoke-keys"
Nov 01 07:21:23.408622 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.408602    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.461112 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.461082    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c6c6bf-serving-cert\""
Nov 01 07:21:23.461225 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.461130    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:21:23.510540 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.510507    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:21:23.510540 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.510538    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 07:21:23.510718 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.510595    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rqph5\""
Nov 01 07:21:23.512701 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.512683    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"]
Nov 01 07:21:23.536047 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.536018    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.536192 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.536051    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.536455 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.536188    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqxw\" (UniqueName: \"kubernetes.io/projected/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-kube-api-access-5vqxw\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.536455 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.536385    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.637991 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.637958    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqxw\" (UniqueName: \"kubernetes.io/projected/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-kube-api-access-5vqxw\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.638619 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.638589    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.639619 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:21:23.639322    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-serving-cert: secret "maas-api-e2e-worker-single-main-c6c6bf-serving-cert" not found
Nov 01 07:21:23.639619 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:21:23.639406    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls podName:fbfc977e-5c17-4ca5-a8e8-c625478c0d3c nodeName:}" failed. No retries permitted until 2026-11-01 07:21:24.139383181 +0000 UTC m=+5721.017356584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls") pod "maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8" (UID: "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c") : secret "maas-api-e2e-worker-single-main-c6c6bf-serving-cert" not found
Nov 01 07:21:23.641251 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.640789    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.648815 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.648570    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.649548 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.649518    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.649744 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.649664    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:23.700095 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:23.700016    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqxw\" (UniqueName: \"kubernetes.io/projected/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-kube-api-access-5vqxw\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:24.153500 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:24.153467    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:24.156078 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:24.156047    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:24.321626 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:24.321600    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:24.697023 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:24.696860    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"]
Nov 01 07:21:24.699943 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:21:24.699907    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfc977e_5c17_4ca5_a8e8_c625478c0d3c.slice/crio-85c27b7670408f2327dbb0275d87b0a6da2d12de1ca2db050cf85c4e178c3779 WatchSource:0}: Error finding container 85c27b7670408f2327dbb0275d87b0a6da2d12de1ca2db050cf85c4e178c3779: Status 404 returned error can't find the container with id 85c27b7670408f2327dbb0275d87b0a6da2d12de1ca2db050cf85c4e178c3779
Nov 01 07:21:24.701307 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:24.701289    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 07:21:25.564216 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:25.564179    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8" event={"ID":"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c","Type":"ContainerStarted","Data":"85c27b7670408f2327dbb0275d87b0a6da2d12de1ca2db050cf85c4e178c3779"}
Nov 01 07:21:26.571383 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:26.571345    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8" event={"ID":"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c","Type":"ContainerStarted","Data":"3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8"}
Nov 01 07:21:26.571809 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:26.571524    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:26.689968 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:26.689918    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8" podStartSLOduration=2.727644213 podStartE2EDuration="3.689904772s" totalImagesPullingTime="962.260559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:21:26.687650727 +0000 UTC m=+5723.565624136" watchObservedRunningTime="2026-11-01 07:21:26.689904772 +0000 UTC m=+5723.567878216"
Nov 01 07:21:30.046933 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:30.046897    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"]
Nov 01 07:21:30.076915 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:30.076886    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8c9jg"]
Nov 01 07:21:31.235121 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:31.235086    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a5eeea-28d7-49fd-bf2f-48a3259cfd67" path="/var/lib/kubelet/pods/c9a5eeea-28d7-49fd-bf2f-48a3259cfd67/volumes"
Nov 01 07:21:31.567675 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:31.567578    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:21:32.581499 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:32.581473    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:21:47.543971 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:21:47.543937    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:21:51.199564 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:21:51.199531    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:22:05.505074 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:05.505043    5510 scope.go:122] "RemoveContainer" containerID="2a656600f5feafc46912b79c6370760c4542b5988008b14d537e88ce0d732a61"
Nov 01 07:22:09.057135 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:09.057100    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"]
Nov 01 07:22:09.106299 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:09.106269    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fklx6"]
Nov 01 07:22:09.234240 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:09.234210    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bc1ef9-082f-42e7-ac60-accf4a9b1982" path="/var/lib/kubelet/pods/76bc1ef9-082f-42e7-ac60-accf4a9b1982/volumes"
Nov 01 07:22:15.924406 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:15.924328    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:22:17.544445 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:22:17.544407    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:22:38.046264 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:38.046229    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"]
Nov 01 07:22:38.095585 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:38.095557    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pzkz8"]
Nov 01 07:22:39.234775 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:39.234745    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2fba914-6237-489a-8741-0c395365cfa7" path="/var/lib/kubelet/pods/b2fba914-6237-489a-8741-0c395365cfa7/volumes"
Nov 01 07:22:45.299010 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:45.298962    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:22:47.544116 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:22:47.544075    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:22:49.256743 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:49.256707    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"]
Nov 01 07:22:49.257276 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:49.256909    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8" podUID="fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" containerName="maas-api" containerID="cri-o://3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8" gracePeriod=30
Nov 01 07:22:50.504392 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.504359    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:22:50.595053 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.594981    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-metrics-certs\") pod \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") "
Nov 01 07:22:50.595195 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.595096    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-kube-api-access-5vqxw\" (UniqueName: \"kubernetes.io/projected/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-kube-api-access-5vqxw\") pod \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") "
Nov 01 07:22:50.595195 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.595122    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls\") pod \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") "
Nov 01 07:22:50.595195 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.595156    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-openshift-service-ca\") pod \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\" (UID: \"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c\") "
Nov 01 07:22:50.595582 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.595555    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-openshift-service-ca" pod "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" (UID: "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:22:50.597086 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.597066    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-metrics-certs" pod "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" (UID: "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:22:50.597284 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.597270    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-kube-api-access-5vqxw" pod "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" (UID: "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c"). InnerVolumeSpecName "kube-api-access-5vqxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:22:50.597401 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.597373    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls" pod "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" (UID: "fbfc977e-5c17-4ca5-a8e8-c625478c0d3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:22:50.695989 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.695948    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:50.695989 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.695978    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:50.695989 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.695989    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vqxw\" (UniqueName: \"kubernetes.io/projected/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-kube-api-access-5vqxw\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:50.695989 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:50.695997    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:51.014686 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.014650    5510 generic.go:356] "Generic (PLEG): container finished" podID="fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" containerID="3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8" exitCode=0
Nov 01 07:22:51.014909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.014703    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"
Nov 01 07:22:51.014909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.014739    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8" event={"ID":"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c","Type":"ContainerDied","Data":"3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8"}
Nov 01 07:22:51.014909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.014854    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8" event={"ID":"fbfc977e-5c17-4ca5-a8e8-c625478c0d3c","Type":"ContainerDied","Data":"85c27b7670408f2327dbb0275d87b0a6da2d12de1ca2db050cf85c4e178c3779"}
Nov 01 07:22:51.014909 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.014886    5510 scope.go:122] "RemoveContainer" containerID="3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8"
Nov 01 07:22:51.024717 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.024700    5510 scope.go:122] "RemoveContainer" containerID="3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8"
Nov 01 07:22:51.025064 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:22:51.025042    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8\": container with ID starting with 3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8 not found: ID does not exist" containerID="3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8"
Nov 01 07:22:51.025148 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.025070    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8"} err="failed to get container status \"3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8\": rpc error: code = NotFound desc = could not find container \"3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8\": container with ID starting with 3ea8059fb761cef5dd358cfb896d99e6bb8fa4ac5db41c00bc22a24d84c2b5e8 not found: ID does not exist"
Nov 01 07:22:51.178291 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.178256    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"]
Nov 01 07:22:51.182037 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.182013    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-m6dw8"]
Nov 01 07:22:51.234560 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:51.234536    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" path="/var/lib/kubelet/pods/fbfc977e-5c17-4ca5-a8e8-c625478c0d3c/volumes"
Nov 01 07:22:59.047939 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:59.047905    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"]
Nov 01 07:22:59.089688 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:59.089638    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ns7fh"]
Nov 01 07:22:59.162049 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:59.162015    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:22:59.234084 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:22:59.234053    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0934cf6-3be6-4d53-ae20-ff21e28ce25f" path="/var/lib/kubelet/pods/e0934cf6-3be6-4d53-ae20-ff21e28ce25f/volumes"
Nov 01 07:23:05.525546 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:05.525510    5510 scope.go:122] "RemoveContainer" containerID="23867d20e5be4fcc92c51f2e784aaa78f42e57b41efedc007a6f681214e06a8e"
Nov 01 07:23:05.534490 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:05.534467    5510 scope.go:122] "RemoveContainer" containerID="ba63f587e14dad06bf2d0687348dc745954ccb49bd8e11d1cb6e2c4c061e9726"
Nov 01 07:23:05.543457 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:05.543433    5510 scope.go:122] "RemoveContainer" containerID="bc77658f24005abe220da51240682bbd883b52c554391797cc580f613445b269"
Nov 01 07:23:16.046880 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:16.046845    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"]
Nov 01 07:23:16.095729 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:16.095703    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cgdg2"]
Nov 01 07:23:17.234769 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:17.234736    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e247f873-438f-4f55-a39e-de3fdf398fc5" path="/var/lib/kubelet/pods/e247f873-438f-4f55-a39e-de3fdf398fc5/volumes"
Nov 01 07:23:17.545231 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:23:17.545146    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:23:34.689103 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:34.689069    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:23:37.662480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.662444    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"]
Nov 01 07:23:37.662967 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.662941    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" containerName="maas-api"
Nov 01 07:23:37.662967 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.662961    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" containerName="maas-api"
Nov 01 07:23:37.663244 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.663040    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfc977e-5c17-4ca5-a8e8-c625478c0d3c" containerName="maas-api"
Nov 01 07:23:37.665801 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.665786    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.717854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.717804    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:23:37.752561 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.752529    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhd87\" (UniqueName: \"kubernetes.io/projected/64480782-a2df-40a5-9de3-f19a18523ded-kube-api-access-nhd87\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.752734 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.752592    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.752734 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.752637    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.767281 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.767249    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:23:37.767281 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.767262    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:23:37.767442 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.767313    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:23:37.768957 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.768939    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"]
Nov 01 07:23:37.853269 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.853237    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhd87\" (UniqueName: \"kubernetes.io/projected/64480782-a2df-40a5-9de3-f19a18523ded-kube-api-access-nhd87\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.853558 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.853514    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.853714 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.853574    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.854305 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.854285    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.856176 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.856150    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.914642 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.914578    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhd87\" (UniqueName: \"kubernetes.io/projected/64480782-a2df-40a5-9de3-f19a18523ded-kube-api-access-nhd87\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:37.978642 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:37.978614    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:38.348987 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:38.348925    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"]
Nov 01 07:23:38.351855 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:23:38.351796    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64480782_a2df_40a5_9de3_f19a18523ded.slice/crio-40ecc7d223eae182b8e9249a9e1d30abbf9744a81029b87b76915a18a198a07e WatchSource:0}: Error finding container 40ecc7d223eae182b8e9249a9e1d30abbf9744a81029b87b76915a18a198a07e: Status 404 returned error can't find the container with id 40ecc7d223eae182b8e9249a9e1d30abbf9744a81029b87b76915a18a198a07e
Nov 01 07:23:39.260150 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:39.260125    5510 generic.go:356] "Generic (PLEG): container finished" podID="64480782-a2df-40a5-9de3-f19a18523ded" containerID="9b5f2f5fd18208df37d45e0d3071ced6938b69b401313c9f01d69dac9405edfa" exitCode=0
Nov 01 07:23:39.260473 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:39.260183    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj" event={"ID":"64480782-a2df-40a5-9de3-f19a18523ded","Type":"ContainerDied","Data":"9b5f2f5fd18208df37d45e0d3071ced6938b69b401313c9f01d69dac9405edfa"}
Nov 01 07:23:39.260473 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:39.260218    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj" event={"ID":"64480782-a2df-40a5-9de3-f19a18523ded","Type":"ContainerStarted","Data":"40ecc7d223eae182b8e9249a9e1d30abbf9744a81029b87b76915a18a198a07e"}
Nov 01 07:23:40.422017 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.421993    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:40.482080 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.482055    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64480782-a2df-40a5-9de3-f19a18523ded-kube-api-access-nhd87\" (UniqueName: \"kubernetes.io/projected/64480782-a2df-40a5-9de3-f19a18523ded-kube-api-access-nhd87\") pod \"64480782-a2df-40a5-9de3-f19a18523ded\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") "
Nov 01 07:23:40.482196 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.482129    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-xks-ca\") pod \"64480782-a2df-40a5-9de3-f19a18523ded\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") "
Nov 01 07:23:40.482250 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.482201    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-openshift-ca\") pod \"64480782-a2df-40a5-9de3-f19a18523ded\" (UID: \"64480782-a2df-40a5-9de3-f19a18523ded\") "
Nov 01 07:23:40.482554 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.482526    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-openshift-ca" pod "64480782-a2df-40a5-9de3-f19a18523ded" (UID: "64480782-a2df-40a5-9de3-f19a18523ded"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:23:40.484256 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.484225    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64480782-a2df-40a5-9de3-f19a18523ded-kube-api-access-nhd87" pod "64480782-a2df-40a5-9de3-f19a18523ded" (UID: "64480782-a2df-40a5-9de3-f19a18523ded"). InnerVolumeSpecName "kube-api-access-nhd87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:23:40.484443 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.484426    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-xks-ca" pod "64480782-a2df-40a5-9de3-f19a18523ded" (UID: "64480782-a2df-40a5-9de3-f19a18523ded"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:23:40.583074 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.583001    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:40.583074 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.583028    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhd87\" (UniqueName: \"kubernetes.io/projected/64480782-a2df-40a5-9de3-f19a18523ded-kube-api-access-nhd87\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:40.583074 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:40.583049    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64480782-a2df-40a5-9de3-f19a18523ded-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:41.270866 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:41.270836    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj" event={"ID":"64480782-a2df-40a5-9de3-f19a18523ded","Type":"ContainerDied","Data":"40ecc7d223eae182b8e9249a9e1d30abbf9744a81029b87b76915a18a198a07e"}
Nov 01 07:23:41.270866 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:41.270860    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"
Nov 01 07:23:41.271163 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:41.270881    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ecc7d223eae182b8e9249a9e1d30abbf9744a81029b87b76915a18a198a07e"
Nov 01 07:23:47.545189 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:23:47.545092    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:23:56.038758 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:56.038727    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"]
Nov 01 07:23:56.087707 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:56.087679    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c6c6bf-cnfpj"]
Nov 01 07:23:57.235244 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:57.235210    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64480782-a2df-40a5-9de3-f19a18523ded" path="/var/lib/kubelet/pods/64480782-a2df-40a5-9de3-f19a18523ded/volumes"
Nov 01 07:23:58.091842 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.091791    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"]
Nov 01 07:23:58.141127 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.141093    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwfb"]
Nov 01 07:23:58.213066 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.213032    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"]
Nov 01 07:23:58.213450 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.213438    5510 cpu_manager.go:414] "containerMap: removing container" podUID="64480782-a2df-40a5-9de3-f19a18523ded" containerName="revoke-keys"
Nov 01 07:23:58.213501 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.213451    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64480782-a2df-40a5-9de3-f19a18523ded" containerName="revoke-keys"
Nov 01 07:23:58.213540 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.213508    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="64480782-a2df-40a5-9de3-f19a18523ded" containerName="revoke-keys"
Nov 01 07:23:58.216663 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.216644    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.269230 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.269203    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:23:58.269616 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.269520    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:23:58.269703 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.269679    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:23:58.275143 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.275088    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"]
Nov 01 07:23:58.335315 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.335284    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:23:58.352465 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.352397    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqnff\" (UniqueName: \"kubernetes.io/projected/280c1232-16ee-4c47-a860-ed59b2587f67-kube-api-access-kqnff\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.352465 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.352430    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.352680 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.352541    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.453706 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.453672    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqnff\" (UniqueName: \"kubernetes.io/projected/280c1232-16ee-4c47-a860-ed59b2587f67-kube-api-access-kqnff\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.453897 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.453714    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.453897 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.453748    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.454370 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.454350    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.456395 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.456379    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.513784 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.513754    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqnff\" (UniqueName: \"kubernetes.io/projected/280c1232-16ee-4c47-a860-ed59b2587f67-kube-api-access-kqnff\") pod \"maas-api-revoke-sub-3d4156f1-vv4nc\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.529211 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.529194    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:23:58.903108 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:58.903070    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"]
Nov 01 07:23:58.905633 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:23:58.905607    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280c1232_16ee_4c47_a860_ed59b2587f67.slice/crio-777a2625e9efc2b3bc77c3cd278aa9bb44604e543f45dc5016511b70984bc3ea WatchSource:0}: Error finding container 777a2625e9efc2b3bc77c3cd278aa9bb44604e543f45dc5016511b70984bc3ea: Status 404 returned error can't find the container with id 777a2625e9efc2b3bc77c3cd278aa9bb44604e543f45dc5016511b70984bc3ea
Nov 01 07:23:59.234600 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:59.234569    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1386725d-8002-4035-b052-0d037c22d1fe" path="/var/lib/kubelet/pods/1386725d-8002-4035-b052-0d037c22d1fe/volumes"
Nov 01 07:23:59.372015 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:59.371981    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc" event={"ID":"280c1232-16ee-4c47-a860-ed59b2587f67","Type":"ContainerStarted","Data":"9c389559d15286d26efbeb91611bc42d880a01f9ec82dcf09250783ac74488ea"}
Nov 01 07:23:59.372493 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:59.372040    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc" event={"ID":"280c1232-16ee-4c47-a860-ed59b2587f67","Type":"ContainerStarted","Data":"777a2625e9efc2b3bc77c3cd278aa9bb44604e543f45dc5016511b70984bc3ea"}
Nov 01 07:23:59.487751 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:23:59.487703    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc" podStartSLOduration=1.48768796 podStartE2EDuration="1.48768796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:23:59.486773583 +0000 UTC m=+5876.364746994" watchObservedRunningTime="2026-11-01 07:23:59.48768796 +0000 UTC m=+5876.365661416"
Nov 01 07:24:00.104519 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:00.104488    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:24:00.378330 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:00.378243    5510 generic.go:356] "Generic (PLEG): container finished" podID="280c1232-16ee-4c47-a860-ed59b2587f67" containerID="9c389559d15286d26efbeb91611bc42d880a01f9ec82dcf09250783ac74488ea" exitCode=0
Nov 01 07:24:00.378760 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:00.378325    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc" event={"ID":"280c1232-16ee-4c47-a860-ed59b2587f67","Type":"ContainerDied","Data":"9c389559d15286d26efbeb91611bc42d880a01f9ec82dcf09250783ac74488ea"}
Nov 01 07:24:01.528394 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.528372    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:24:01.686001 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.685925    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-xks-ca\") pod \"280c1232-16ee-4c47-a860-ed59b2587f67\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") "
Nov 01 07:24:01.686169 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.686038    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280c1232-16ee-4c47-a860-ed59b2587f67-kube-api-access-kqnff\" (UniqueName: \"kubernetes.io/projected/280c1232-16ee-4c47-a860-ed59b2587f67-kube-api-access-kqnff\") pod \"280c1232-16ee-4c47-a860-ed59b2587f67\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") "
Nov 01 07:24:01.686214 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.686176    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-openshift-ca\") pod \"280c1232-16ee-4c47-a860-ed59b2587f67\" (UID: \"280c1232-16ee-4c47-a860-ed59b2587f67\") "
Nov 01 07:24:01.686491 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.686463    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-openshift-ca" pod "280c1232-16ee-4c47-a860-ed59b2587f67" (UID: "280c1232-16ee-4c47-a860-ed59b2587f67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:24:01.688248 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.688225    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-xks-ca" pod "280c1232-16ee-4c47-a860-ed59b2587f67" (UID: "280c1232-16ee-4c47-a860-ed59b2587f67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:24:01.688346 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.688249    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280c1232-16ee-4c47-a860-ed59b2587f67-kube-api-access-kqnff" pod "280c1232-16ee-4c47-a860-ed59b2587f67" (UID: "280c1232-16ee-4c47-a860-ed59b2587f67"). InnerVolumeSpecName "kube-api-access-kqnff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:24:01.787709 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.787675    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqnff\" (UniqueName: \"kubernetes.io/projected/280c1232-16ee-4c47-a860-ed59b2587f67-kube-api-access-kqnff\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.787709 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.787705    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.787953 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:01.787719    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/280c1232-16ee-4c47-a860-ed59b2587f67-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:02.388884 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:02.388858    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"
Nov 01 07:24:02.388884 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:02.388875    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc" event={"ID":"280c1232-16ee-4c47-a860-ed59b2587f67","Type":"ContainerDied","Data":"777a2625e9efc2b3bc77c3cd278aa9bb44604e543f45dc5016511b70984bc3ea"}
Nov 01 07:24:02.389111 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:02.388930    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="777a2625e9efc2b3bc77c3cd278aa9bb44604e543f45dc5016511b70984bc3ea"
Nov 01 07:24:05.575776 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:05.575741    5510 scope.go:122] "RemoveContainer" containerID="53f1f500373538dd0a358dbf6777613262dcb958e6c53d077c7f3e80b38bee3d"
Nov 01 07:24:05.585007 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:05.584988    5510 scope.go:122] "RemoveContainer" containerID="23c7a03636c150a10601183b37cc0c517a90a43959b07eb8da897d452e4d78d5"
Nov 01 07:24:17.544527 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:24:17.544494    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:24:17.572770 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:17.572746    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:24:21.861710 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.861672    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"]
Nov 01 07:24:21.862226 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.862126    5510 cpu_manager.go:414] "containerMap: removing container" podUID="280c1232-16ee-4c47-a860-ed59b2587f67" containerName="revoke-keys"
Nov 01 07:24:21.862226 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.862138    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280c1232-16ee-4c47-a860-ed59b2587f67" containerName="revoke-keys"
Nov 01 07:24:21.862226 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.862218    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="280c1232-16ee-4c47-a860-ed59b2587f67" containerName="revoke-keys"
Nov 01 07:24:21.866695 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.866679    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:21.923641 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.923608    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"]
Nov 01 07:24:21.924988 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.924962    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zz8f\" (UniqueName: \"kubernetes.io/projected/53bba979-5f83-4ac6-beea-56290db32175-kube-api-access-8zz8f\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:21.925192 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.925028    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bba979-5f83-4ac6-beea-56290db32175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:21.925192 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.925175    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bba979-5f83-4ac6-beea-56290db32175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:21.971073 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.971040    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:24:21.971073 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.971056    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:24:21.971073 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.971043    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:24:21.971309 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:21.971088    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cf2w\""
Nov 01 07:24:22.026212 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:22.026183    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zz8f\" (UniqueName: \"kubernetes.io/projected/53bba979-5f83-4ac6-beea-56290db32175-kube-api-access-8zz8f\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:22.026360 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:22.026226    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bba979-5f83-4ac6-beea-56290db32175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:22.026360 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:22.026244    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bba979-5f83-4ac6-beea-56290db32175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:22.026782 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:22.026763    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bba979-5f83-4ac6-beea-56290db32175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:22.028658 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:22.028637    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bba979-5f83-4ac6-beea-56290db32175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:22.086948 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:22.086923    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zz8f\" (UniqueName: \"kubernetes.io/projected/53bba979-5f83-4ac6-beea-56290db32175-kube-api-access-8zz8f\") pod \"maas-api-revoke-sub-d2bc93bc-57fjp\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:22.185752 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:22.185682    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:22.557138 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:22.557089    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"]
Nov 01 07:24:22.559897 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:24:22.559871    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bba979_5f83_4ac6_beea_56290db32175.slice/crio-137afbaa0cd4207c324e48b0e8f99fa2f71fd1296fbfb820ef580f04aba790ba WatchSource:0}: Error finding container 137afbaa0cd4207c324e48b0e8f99fa2f71fd1296fbfb820ef580f04aba790ba: Status 404 returned error can't find the container with id 137afbaa0cd4207c324e48b0e8f99fa2f71fd1296fbfb820ef580f04aba790ba
Nov 01 07:24:23.514531 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:23.514490    5510 generic.go:356] "Generic (PLEG): container finished" podID="53bba979-5f83-4ac6-beea-56290db32175" containerID="7cb3199659925dea78655ac652c86f521d72002043e213225432b8c0f74a8bf5" exitCode=0
Nov 01 07:24:23.514901 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:23.514617    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp" event={"ID":"53bba979-5f83-4ac6-beea-56290db32175","Type":"ContainerDied","Data":"7cb3199659925dea78655ac652c86f521d72002043e213225432b8c0f74a8bf5"}
Nov 01 07:24:23.514901 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:23.514667    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp" event={"ID":"53bba979-5f83-4ac6-beea-56290db32175","Type":"ContainerStarted","Data":"137afbaa0cd4207c324e48b0e8f99fa2f71fd1296fbfb820ef580f04aba790ba"}
Nov 01 07:24:24.661043 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.661021    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:24.752958 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.752935    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bba979-5f83-4ac6-beea-56290db32175-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bba979-5f83-4ac6-beea-56290db32175-maas-api-openshift-ca\") pod \"53bba979-5f83-4ac6-beea-56290db32175\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") "
Nov 01 07:24:24.753124 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.753029    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53bba979-5f83-4ac6-beea-56290db32175-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bba979-5f83-4ac6-beea-56290db32175-maas-api-xks-ca\") pod \"53bba979-5f83-4ac6-beea-56290db32175\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") "
Nov 01 07:24:24.753176 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.753126    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bba979-5f83-4ac6-beea-56290db32175-kube-api-access-8zz8f\" (UniqueName: \"kubernetes.io/projected/53bba979-5f83-4ac6-beea-56290db32175-kube-api-access-8zz8f\") pod \"53bba979-5f83-4ac6-beea-56290db32175\" (UID: \"53bba979-5f83-4ac6-beea-56290db32175\") "
Nov 01 07:24:24.753331 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.753304    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bba979-5f83-4ac6-beea-56290db32175-maas-api-openshift-ca" pod "53bba979-5f83-4ac6-beea-56290db32175" (UID: "53bba979-5f83-4ac6-beea-56290db32175"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:24:24.755149 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.755127    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bba979-5f83-4ac6-beea-56290db32175-maas-api-xks-ca" pod "53bba979-5f83-4ac6-beea-56290db32175" (UID: "53bba979-5f83-4ac6-beea-56290db32175"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:24:24.755260 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.755231    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bba979-5f83-4ac6-beea-56290db32175-kube-api-access-8zz8f" pod "53bba979-5f83-4ac6-beea-56290db32175" (UID: "53bba979-5f83-4ac6-beea-56290db32175"). InnerVolumeSpecName "kube-api-access-8zz8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:24:24.854756 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.854686    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zz8f\" (UniqueName: \"kubernetes.io/projected/53bba979-5f83-4ac6-beea-56290db32175-kube-api-access-8zz8f\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:24.854756 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.854711    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bba979-5f83-4ac6-beea-56290db32175-maas-api-openshift-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:24.854756 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:24.854723    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bba979-5f83-4ac6-beea-56290db32175-maas-api-xks-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:25.525317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:25.525234    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"
Nov 01 07:24:25.525317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:25.525256    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp" event={"ID":"53bba979-5f83-4ac6-beea-56290db32175","Type":"ContainerDied","Data":"137afbaa0cd4207c324e48b0e8f99fa2f71fd1296fbfb820ef580f04aba790ba"}
Nov 01 07:24:25.525317 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:25.525301    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="137afbaa0cd4207c324e48b0e8f99fa2f71fd1296fbfb820ef580f04aba790ba"
Nov 01 07:24:26.911588 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:26.911554    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:24:34.759879 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.759820    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"]
Nov 01 07:24:34.760377 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.760358    5510 cpu_manager.go:414] "containerMap: removing container" podUID="53bba979-5f83-4ac6-beea-56290db32175" containerName="revoke-keys"
Nov 01 07:24:34.760438 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.760378    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bba979-5f83-4ac6-beea-56290db32175" containerName="revoke-keys"
Nov 01 07:24:34.760480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.760466    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bba979-5f83-4ac6-beea-56290db32175" containerName="revoke-keys"
Nov 01 07:24:34.763689 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.763673    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.815926 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.815899    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:24:34.850014 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.849982    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjg9p\" (UniqueName: \"kubernetes.io/projected/c4891c69-b136-4f12-ba04-31490eaa66b8-kube-api-access-kjg9p\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.850159 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.850022    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-maas-api-tls\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.850159 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.850085    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-metrics-certs\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.850244 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.850185    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4891c69-b136-4f12-ba04-31490eaa66b8-openshift-service-ca\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.865331 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.865305    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:24:34.865758 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.865737    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 07:24:34.865934 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.865820    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 07:24:34.865934 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.865891    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rqph5\""
Nov 01 07:24:34.868069 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.868048    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"]
Nov 01 07:24:34.951463 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.951434    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4891c69-b136-4f12-ba04-31490eaa66b8-openshift-service-ca\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.951671 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.951484    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjg9p\" (UniqueName: \"kubernetes.io/projected/c4891c69-b136-4f12-ba04-31490eaa66b8-kube-api-access-kjg9p\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.951671 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.951510    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-maas-api-tls\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.951671 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.951537    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-metrics-certs\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.952236 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.952210    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4891c69-b136-4f12-ba04-31490eaa66b8-openshift-service-ca\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.954216 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.954183    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-metrics-certs\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:34.954315 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:34.954296    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-maas-api-tls\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:35.011783 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:35.011729    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjg9p\" (UniqueName: \"kubernetes.io/projected/c4891c69-b136-4f12-ba04-31490eaa66b8-kube-api-access-kjg9p\") pod \"maas-api-bdb66f487-stzhr\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:35.080764 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:35.080738    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:35.457564 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:35.457538    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"]
Nov 01 07:24:35.459652 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:24:35.459627    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4891c69_b136_4f12_ba04_31490eaa66b8.slice/crio-322a00d3165bc2c87e7b30728cb4e30a701b171cdd9b753e1ff2623d21ece261 WatchSource:0}: Error finding container 322a00d3165bc2c87e7b30728cb4e30a701b171cdd9b753e1ff2623d21ece261: Status 404 returned error can't find the container with id 322a00d3165bc2c87e7b30728cb4e30a701b171cdd9b753e1ff2623d21ece261
Nov 01 07:24:35.580285 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:35.580246    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr" event={"ID":"c4891c69-b136-4f12-ba04-31490eaa66b8","Type":"ContainerStarted","Data":"322a00d3165bc2c87e7b30728cb4e30a701b171cdd9b753e1ff2623d21ece261"}
Nov 01 07:24:36.586689 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:36.586656    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr" event={"ID":"c4891c69-b136-4f12-ba04-31490eaa66b8","Type":"ContainerStarted","Data":"094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e"}
Nov 01 07:24:36.587154 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:36.586718    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:36.703609 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:36.703451    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr" podStartSLOduration=1.796362902 podStartE2EDuration="2.703437371s" totalImagesPullingTime="907.074469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:24:36.703036513 +0000 UTC m=+5913.581009924" watchObservedRunningTime="2026-11-01 07:24:36.703437371 +0000 UTC m=+5913.581410787"
Nov 01 07:24:42.596133 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:42.596105    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:43.560805 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.560767    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"]
Nov 01 07:24:43.564615 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.564593    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.668362 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.668329    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"]
Nov 01 07:24:43.732780 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.732744    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b0a7d-f7db-46a9-ac10-48176f8531b4-openshift-service-ca\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.732780 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.732783    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77b0a7d-f7db-46a9-ac10-48176f8531b4-maas-api-tls\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.733018 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.732891    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqsw\" (UniqueName: \"kubernetes.io/projected/e77b0a7d-f7db-46a9-ac10-48176f8531b4-kube-api-access-pxqsw\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.733018 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.732973    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77b0a7d-f7db-46a9-ac10-48176f8531b4-metrics-certs\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.834441 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.834373    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77b0a7d-f7db-46a9-ac10-48176f8531b4-metrics-certs\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.834441 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.834408    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b0a7d-f7db-46a9-ac10-48176f8531b4-openshift-service-ca\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.834441 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.834427    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77b0a7d-f7db-46a9-ac10-48176f8531b4-maas-api-tls\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.834645 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.834468    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqsw\" (UniqueName: \"kubernetes.io/projected/e77b0a7d-f7db-46a9-ac10-48176f8531b4-kube-api-access-pxqsw\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.835060 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.835027    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b0a7d-f7db-46a9-ac10-48176f8531b4-openshift-service-ca\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.837069 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.837048    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77b0a7d-f7db-46a9-ac10-48176f8531b4-metrics-certs\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.837187 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.837091    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77b0a7d-f7db-46a9-ac10-48176f8531b4-maas-api-tls\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:43.895756 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:43.895731    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqsw\" (UniqueName: \"kubernetes.io/projected/e77b0a7d-f7db-46a9-ac10-48176f8531b4-kube-api-access-pxqsw\") pod \"maas-api-66f65f7fd7-9x6kz\" (UID: \"e77b0a7d-f7db-46a9-ac10-48176f8531b4\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:44.177588 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:44.177506    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:44.550117 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:44.550090    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"]
Nov 01 07:24:44.552928 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:24:44.552903    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77b0a7d_f7db_46a9_ac10_48176f8531b4.slice/crio-6c7d1e9f9490dfe622750c1243dc7e2a35eba1742fea7c195beb4dcc21f94cbc WatchSource:0}: Error finding container 6c7d1e9f9490dfe622750c1243dc7e2a35eba1742fea7c195beb4dcc21f94cbc: Status 404 returned error can't find the container with id 6c7d1e9f9490dfe622750c1243dc7e2a35eba1742fea7c195beb4dcc21f94cbc
Nov 01 07:24:44.630586 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:44.630554    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz" event={"ID":"e77b0a7d-f7db-46a9-ac10-48176f8531b4","Type":"ContainerStarted","Data":"6c7d1e9f9490dfe622750c1243dc7e2a35eba1742fea7c195beb4dcc21f94cbc"}
Nov 01 07:24:45.637104 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:45.637002    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz" event={"ID":"e77b0a7d-f7db-46a9-ac10-48176f8531b4","Type":"ContainerStarted","Data":"1bd1aabf8f657caab495a5938064ad7708e7ea16e60c95779a035dd47d896c60"}
Nov 01 07:24:45.637477 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:45.637195    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:45.754843 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:45.754779    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz" podStartSLOduration=1.944093625 podStartE2EDuration="2.754766215s" totalImagesPullingTime="810.67259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:24:45.752973666 +0000 UTC m=+5922.630947076" watchObservedRunningTime="2026-11-01 07:24:45.754766215 +0000 UTC m=+5922.632739625"
Nov 01 07:24:47.057535 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:47.057482    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"]
Nov 01 07:24:47.106407 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:47.106376    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8lsm"]
Nov 01 07:24:47.234380 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:47.234347    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32577a4e-9529-4489-9312-711ede5fa50e" path="/var/lib/kubelet/pods/32577a4e-9529-4489-9312-711ede5fa50e/volumes"
Nov 01 07:24:47.544593 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:24:47.544547    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:24:51.646906 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:51.646874    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-9x6kz"
Nov 01 07:24:51.817043 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:51.817015    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"]
Nov 01 07:24:51.817294 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:51.817269    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr" podUID="c4891c69-b136-4f12-ba04-31490eaa66b8" containerName="maas-api" containerID="cri-o://094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e" gracePeriod=30
Nov 01 07:24:52.076406 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.076382    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:52.107864 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.107817    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4891c69-b136-4f12-ba04-31490eaa66b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4891c69-b136-4f12-ba04-31490eaa66b8-openshift-service-ca\") pod \"c4891c69-b136-4f12-ba04-31490eaa66b8\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") "
Nov 01 07:24:52.108007 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.107905    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-maas-api-tls\") pod \"c4891c69-b136-4f12-ba04-31490eaa66b8\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") "
Nov 01 07:24:52.108007 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.107943    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4891c69-b136-4f12-ba04-31490eaa66b8-kube-api-access-kjg9p\" (UniqueName: \"kubernetes.io/projected/c4891c69-b136-4f12-ba04-31490eaa66b8-kube-api-access-kjg9p\") pod \"c4891c69-b136-4f12-ba04-31490eaa66b8\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") "
Nov 01 07:24:52.108007 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.108006    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-metrics-certs\") pod \"c4891c69-b136-4f12-ba04-31490eaa66b8\" (UID: \"c4891c69-b136-4f12-ba04-31490eaa66b8\") "
Nov 01 07:24:52.108194 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.108129    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4891c69-b136-4f12-ba04-31490eaa66b8-openshift-service-ca" pod "c4891c69-b136-4f12-ba04-31490eaa66b8" (UID: "c4891c69-b136-4f12-ba04-31490eaa66b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:24:52.108371 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.108353    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4891c69-b136-4f12-ba04-31490eaa66b8-openshift-service-ca\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:52.110201 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.110180    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-maas-api-tls" pod "c4891c69-b136-4f12-ba04-31490eaa66b8" (UID: "c4891c69-b136-4f12-ba04-31490eaa66b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:24:52.110280 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.110200    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-metrics-certs" pod "c4891c69-b136-4f12-ba04-31490eaa66b8" (UID: "c4891c69-b136-4f12-ba04-31490eaa66b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:24:52.110280 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.110213    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4891c69-b136-4f12-ba04-31490eaa66b8-kube-api-access-kjg9p" pod "c4891c69-b136-4f12-ba04-31490eaa66b8" (UID: "c4891c69-b136-4f12-ba04-31490eaa66b8"). InnerVolumeSpecName "kube-api-access-kjg9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:24:52.209454 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.209428    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-maas-api-tls\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:52.209454 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.209453    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjg9p\" (UniqueName: \"kubernetes.io/projected/c4891c69-b136-4f12-ba04-31490eaa66b8-kube-api-access-kjg9p\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:52.209645 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.209466    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4891c69-b136-4f12-ba04-31490eaa66b8-metrics-certs\") on node \"ip-10-0-138-190.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:52.676157 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.676056    5510 generic.go:356] "Generic (PLEG): container finished" podID="c4891c69-b136-4f12-ba04-31490eaa66b8" containerID="094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e" exitCode=0
Nov 01 07:24:52.676157 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.676129    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"
Nov 01 07:24:52.676694 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.676151    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr" event={"ID":"c4891c69-b136-4f12-ba04-31490eaa66b8","Type":"ContainerDied","Data":"094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e"}
Nov 01 07:24:52.676694 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.676240    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bdb66f487-stzhr" event={"ID":"c4891c69-b136-4f12-ba04-31490eaa66b8","Type":"ContainerDied","Data":"322a00d3165bc2c87e7b30728cb4e30a701b171cdd9b753e1ff2623d21ece261"}
Nov 01 07:24:52.676694 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.676263    5510 scope.go:122] "RemoveContainer" containerID="094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e"
Nov 01 07:24:52.686134 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.686119    5510 scope.go:122] "RemoveContainer" containerID="094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e"
Nov 01 07:24:52.686397 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:24:52.686380    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e\": container with ID starting with 094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e not found: ID does not exist" containerID="094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e"
Nov 01 07:24:52.686477 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.686409    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e"} err="failed to get container status \"094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e\": rpc error: code = NotFound desc = could not find container \"094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e\": container with ID starting with 094805e56016e70f4142fd6e9635ec4a768e8bd9346eaaf5efc76e0fffd9b37e not found: ID does not exist"
Nov 01 07:24:52.840489 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.840459    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"]
Nov 01 07:24:52.889899 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:52.889867    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-stzhr"]
Nov 01 07:24:53.236267 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:53.236233    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4891c69-b136-4f12-ba04-31490eaa66b8" path="/var/lib/kubelet/pods/c4891c69-b136-4f12-ba04-31490eaa66b8/volumes"
Nov 01 07:24:57.394747 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:57.390211    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:24:58.047424 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:58.047375    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"]
Nov 01 07:24:58.096096 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:58.096051    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m77zv"]
Nov 01 07:24:59.254337 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:24:59.254301    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ec81c2-7cc9-49cd-a827-90204ef95311" path="/var/lib/kubelet/pods/a8ec81c2-7cc9-49cd-a827-90204ef95311/volumes"
Nov 01 07:25:04.561719 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:25:04.561096    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:25:04.561719 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:25:04.561176    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:25:05.609798 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:25:05.609767    5510 scope.go:122] "RemoveContainer" containerID="b34aa9b1922d597d7e5916f3a1ad826329d7fb320072629ec472c37fbb2192ce"
Nov 01 07:25:05.618476 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:25:05.618453    5510 scope.go:122] "RemoveContainer" containerID="0c9bf35b5288ed20876be908410a03fb5220278435b1a8b3a6dab097af15b77c"
Nov 01 07:25:05.626638 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:25:05.626620    5510 scope.go:122] "RemoveContainer" containerID="2ddc2f9c1ea3cbb5ef521b25f1442fab5a9239dd6b653af871b7936b069ed2c0"
Nov 01 07:25:17.544014 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:25:17.543931    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:25:32.788568 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:25:32.788529    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:25:45.187424 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:25:45.187389    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:25:47.544475 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:25:47.544434    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:26:05.658256 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:26:05.658222    5510 scope.go:122] "RemoveContainer" containerID="aa8d511869574c10e25bf5e4036189f9cc6555a778fdc2c98aab83333f46a56d"
Nov 01 07:26:05.667245 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:26:05.667228    5510 scope.go:122] "RemoveContainer" containerID="5fd453cb6130f1d13262b25b30cdb07d5cb9bd6c8a9729bc6e1ee0dffa837657"
Nov 01 07:26:08.195665 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:26:08.195626    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:26:14.274715 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:26:14.274681    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal" podUID="784c45e8dd43338b3f3bbbd833ce9232" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 01 07:26:17.559854 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:26:17.559594    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:26:20.476713 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:26:20.476683    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:26:47.545385 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:26:47.545301    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:26:49.603253 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:26:49.603216    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:27:05.690373 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:05.690343    5510 scope.go:122] "RemoveContainer" containerID="3da0054e8a643da603e59d7b486e8bddcc9c47e794308ce031825333c0467cf5"
Nov 01 07:27:10.287174 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:10.287135    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:27:17.558309 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:27:17.558259    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:27:19.455854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:19.454919    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6h48b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:27:19.455854 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:19.454983    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6h48b" podUID="ceddee3f-6781-46eb-ab9b-9c9eb267d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:27:19.563567 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:19.563510    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-plj6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:27:19.563807 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:19.563597    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-plj6l" podUID="66c1fd2b-2b0b-40fc-8e19-fcb2602498a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:27:19.563807 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:19.563662    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dtgh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:27:19.563807 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:19.563684    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dtgh2" podUID="25d6bee8-e720-4cda-bed6-aeba98f0b113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:27:22.995293 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:22.995261    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:27:32.384998 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:32.384966    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:27:47.545078 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:27:47.545043    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:27:55.198572 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:27:55.198527    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:28:05.905021 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:28:05.904986    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4wws"]
Nov 01 07:28:17.544077 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:28:17.543994    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:28:47.544312 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:28:47.544271    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:28:56.411105 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:28:56.411074    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7684cd596b-68gdk_05da642c-8901-434c-859a-ad7386a0302c/manager/0.log"
Nov 01 07:28:57.005308 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:28:57.005276    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2vn9w_d2201b73-9fec-485b-ba93-ef6a56393bca/manager/0.log"
Nov 01 07:29:02.046421 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:02.046376    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"]
Nov 01 07:29:02.095629 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:02.095589    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv4nc"]
Nov 01 07:29:03.233911 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:03.233864    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280c1232-16ee-4c47-a860-ed59b2587f67" path="/var/lib/kubelet/pods/280c1232-16ee-4c47-a860-ed59b2587f67/volumes"
Nov 01 07:29:09.798615 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:09.798583    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-66f65f7fd7-9x6kz_e77b0a7d-f7db-46a9-ac10-48176f8531b4/maas-api/0.log"
Nov 01 07:29:11.223657 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:11.223623    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-57fjp_53bba979-5f83-4ac6-beea-56290db32175/revoke-keys/0.log"
Nov 01 07:29:17.543811 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:29:17.543770    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:29:24.918416 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:24.918385    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l458f_7e89c29e-4c5c-4b3b-91bc-33014e337036/kuadrant-console-plugin/0.log"
Nov 01 07:29:25.047640 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:25.047610    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"]
Nov 01 07:29:25.096821 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:25.096787    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-57fjp"]
Nov 01 07:29:25.234943 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:25.234912    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bba979-5f83-4ac6-beea-56290db32175" path="/var/lib/kubelet/pods/53bba979-5f83-4ac6-beea-56290db32175/volumes"
Nov 01 07:29:25.600052 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:25.599976    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f4wws_def3f09f-380f-4342-9f91-740df732a7fe/limitador/0.log"
Nov 01 07:29:37.137121 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:37.137091    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-fphbk_a38250d9-1ebb-43d3-98ff-1b8c19984e8c/discovery/0.log"
Nov 01 07:29:37.882520 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:37.882494    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6h48b_ceddee3f-6781-46eb-ab9b-9c9eb267d18b/istio-proxy/0.log"
Nov 01 07:29:38.388660 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:38.388625    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-dtgh2_25d6bee8-e720-4cda-bed6-aeba98f0b113/istio-proxy/0.log"
Nov 01 07:29:38.862050 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:38.862018    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-f9ttw_47d488d9-60bc-480d-b851-cea9d9a4edb7/istio-proxy/0.log"
Nov 01 07:29:39.317072 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:39.317046    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-plj6l_66c1fd2b-2b0b-40fc-8e19-fcb2602498a4/istio-proxy/0.log"
Nov 01 07:29:47.543482 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:29:47.543398    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:29:50.607168 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:50.607137    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-644bf9c6bb-8d6wr_be361928-05c7-4596-879d-d2cfe79eabcf/payload-pre-processing/0.log"
Nov 01 07:29:50.833092 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:50.833065    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-68ccf99fdf-bbf6f_8e9f6834-d1a0-415b-a8fe-def794867e99/payload-processing/0.log"
Nov 01 07:29:52.358409 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:52.358376    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-4p5kf_0285b55a-dfd7-4f9c-a4c2-8acad8e49760/main/0.log"
Nov 01 07:29:52.586151 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:52.586113    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9949x_4d534516-5a40-48ba-8f09-6ac5cda95748/main/0.log"
Nov 01 07:29:53.077333 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:53.077307    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t57bv_9ade8493-4326-41f7-8be6-202a3f237998/main/0.log"
Nov 01 07:29:53.533805 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:29:53.533758    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-g5zdp_b96ce760-802d-4382-9c01-8d8a588a9810/main/0.log"
Nov 01 07:30:05.724191 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:30:05.724155    5510 scope.go:122] "RemoveContainer" containerID="9c389559d15286d26efbeb91611bc42d880a01f9ec82dcf09250783ac74488ea"
Nov 01 07:30:05.733013 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:30:05.732991    5510 scope.go:122] "RemoveContainer" containerID="9b5f2f5fd18208df37d45e0d3071ced6938b69b401313c9f01d69dac9405edfa"
Nov 01 07:30:17.544809 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:30:17.544771    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:30:47.544451 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:30:47.544406    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:31:05.755780 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:05.755750    5510 scope.go:122] "RemoveContainer" containerID="7cb3199659925dea78655ac652c86f521d72002043e213225432b8c0f74a8bf5"
Nov 01 07:31:12.555775 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:12.555734    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7684cd596b-68gdk_05da642c-8901-434c-859a-ad7386a0302c/manager/0.log"
Nov 01 07:31:13.154852 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:13.154811    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2vn9w_d2201b73-9fec-485b-ba93-ef6a56393bca/manager/0.log"
Nov 01 07:31:14.014934 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:14.014823    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r5qzs_3bf4eb38-7ce4-45fe-8c38-0d8907a8861f/global-pull-secret-syncer/0.log"
Nov 01 07:31:16.331962 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:16.331922    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jmphx_76bf1b5b-01e1-4f37-b0a9-67a64366ecca/konnectivity-agent/0.log"
Nov 01 07:31:17.544160 ip-10-0-138-190 kubenswrapper[5510]: E1101 07:31:17.544125    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pzwxh" containerName="extractor"
Nov 01 07:31:17.565813 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:17.565780    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-190.us-west-1.compute.internal_784c45e8dd43338b3f3bbbd833ce9232/haproxy/0.log"
Nov 01 07:31:19.367462 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:19.367434    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7684cd596b-68gdk_05da642c-8901-434c-859a-ad7386a0302c/manager/0.log"
Nov 01 07:31:19.815665 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:19.815637    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2vn9w_d2201b73-9fec-485b-ba93-ef6a56393bca/manager/0.log"
Nov 01 07:31:23.547926 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.547888    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"]
Nov 01 07:31:23.548408 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.548322    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c4891c69-b136-4f12-ba04-31490eaa66b8" containerName="maas-api"
Nov 01 07:31:23.548408 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.548338    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4891c69-b136-4f12-ba04-31490eaa66b8" containerName="maas-api"
Nov 01 07:31:23.548550 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.548427    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4891c69-b136-4f12-ba04-31490eaa66b8" containerName="maas-api"
Nov 01 07:31:23.552946 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.552926    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.604935 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.604911    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s25q4\"/\"default-dockercfg-9wmvv\""
Nov 01 07:31:23.604935 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.604918    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s25q4\"/\"openshift-service-ca.crt\""
Nov 01 07:31:23.654344 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.654317    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s25q4\"/\"kube-root-ca.crt\""
Nov 01 07:31:23.655944 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.655922    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"]
Nov 01 07:31:23.701218 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.701187    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-proc\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.701384 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.701233    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-podres\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.701384 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.701252    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-sys\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.701384 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.701306    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-lib-modules\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.701576 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.701450    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmkwm\" (UniqueName: \"kubernetes.io/projected/7caef923-f9db-481a-bfa6-a39ce9bece4a-kube-api-access-xmkwm\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803050 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.802961    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkwm\" (UniqueName: \"kubernetes.io/projected/7caef923-f9db-481a-bfa6-a39ce9bece4a-kube-api-access-xmkwm\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803231 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.803052    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-proc\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803231 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.803100    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-podres\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803231 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.803121    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-sys\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803231 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.803147    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-lib-modules\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803231 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.803172    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-proc\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803231 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.803177    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-podres\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803231 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.803215    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-sys\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.803497 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.803266    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7caef923-f9db-481a-bfa6-a39ce9bece4a-lib-modules\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.864158 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.864127    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkwm\" (UniqueName: \"kubernetes.io/projected/7caef923-f9db-481a-bfa6-a39ce9bece4a-kube-api-access-xmkwm\") pod \"perf-node-gather-daemonset-vblnj\" (UID: \"7caef923-f9db-481a-bfa6-a39ce9bece4a\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:23.864957 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:23.864942    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:24.238683 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:24.238658    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"]
Nov 01 07:31:24.241385 ip-10-0-138-190 kubenswrapper[5510]: W1101 07:31:24.241360    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7caef923_f9db_481a_bfa6_a39ce9bece4a.slice/crio-f00e33746fbc8b31b855c3e40ea07ac7c1dc9c978b35f1b72e4b3a454beccfe8 WatchSource:0}: Error finding container f00e33746fbc8b31b855c3e40ea07ac7c1dc9c978b35f1b72e4b3a454beccfe8: Status 404 returned error can't find the container with id f00e33746fbc8b31b855c3e40ea07ac7c1dc9c978b35f1b72e4b3a454beccfe8
Nov 01 07:31:24.887290 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:24.887255    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj" event={"ID":"7caef923-f9db-481a-bfa6-a39ce9bece4a","Type":"ContainerStarted","Data":"6654c90faa0496d09fd1d164356a8084803f023c1375c7a187b8e6e91ded7f30"}
Nov 01 07:31:24.887710 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:24.887333    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj" event={"ID":"7caef923-f9db-481a-bfa6-a39ce9bece4a","Type":"ContainerStarted","Data":"f00e33746fbc8b31b855c3e40ea07ac7c1dc9c978b35f1b72e4b3a454beccfe8"}
Nov 01 07:31:24.887710 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:24.887381    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:25.002643 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:25.002591    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj" podStartSLOduration=2.002577474 podStartE2EDuration="2.002577474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 07:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:31:25.001562319 +0000 UTC m=+6321.879535731" watchObservedRunningTime="2026-11-01 07:31:25.002577474 +0000 UTC m=+6321.880550883"
Nov 01 07:31:25.039923 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:25.039898    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l458f_7e89c29e-4c5c-4b3b-91bc-33014e337036/kuadrant-console-plugin/0.log"
Nov 01 07:31:25.571295 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:25.571266    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7684cd596b-68gdk_05da642c-8901-434c-859a-ad7386a0302c/manager/0.log"
Nov 01 07:31:25.884046 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:25.883968    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f4wws_def3f09f-380f-4342-9f91-740df732a7fe/limitador/0.log"
Nov 01 07:31:26.014477 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:26.014448    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2vn9w_d2201b73-9fec-485b-ba93-ef6a56393bca/manager/0.log"
Nov 01 07:31:30.902186 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:30.902159    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-vblnj"
Nov 01 07:31:32.018790 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:32.018757    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-fphbk_a38250d9-1ebb-43d3-98ff-1b8c19984e8c/discovery/0.log"
Nov 01 07:31:32.366648 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:32.366572    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k8w5m_8c92dc00-2bba-4345-a8c3-b36c72a24056/kube-state-metrics/0.log"
Nov 01 07:31:32.562745 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:32.562716    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6h48b_ceddee3f-6781-46eb-ab9b-9c9eb267d18b/istio-proxy/0.log"
Nov 01 07:31:32.780015 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:32.779984    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k8w5m_8c92dc00-2bba-4345-a8c3-b36c72a24056/kube-rbac-proxy-main/0.log"
Nov 01 07:31:32.990280 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:32.990251    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-dtgh2_25d6bee8-e720-4cda-bed6-aeba98f0b113/istio-proxy/0.log"
Nov 01 07:31:33.196661 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:33.196582    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k8w5m_8c92dc00-2bba-4345-a8c3-b36c72a24056/kube-rbac-proxy-self/0.log"
Nov 01 07:31:33.417985 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:33.417958    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-f9ttw_47d488d9-60bc-480d-b851-cea9d9a4edb7/istio-proxy/0.log"
Nov 01 07:31:33.841740 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:33.841710    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-plj6l_66c1fd2b-2b0b-40fc-8e19-fcb2602498a4/istio-proxy/0.log"
Nov 01 07:31:34.906362 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:34.906332    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-644bf9c6bb-8d6wr_be361928-05c7-4596-879d-d2cfe79eabcf/payload-pre-processing/0.log"
Nov 01 07:31:35.154205 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:35.154175    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-68ccf99fdf-bbf6f_8e9f6834-d1a0-415b-a8fe-def794867e99/payload-processing/0.log"
Nov 01 07:31:37.050504 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:37.050480    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7sbw_c39c479a-5489-41fe-b2e4-28f033f0e021/node-exporter/0.log"
Nov 01 07:31:37.467480 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:37.467455    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7sbw_c39c479a-5489-41fe-b2e4-28f033f0e021/kube-rbac-proxy/0.log"
Nov 01 07:31:37.891101 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:37.891008    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7sbw_c39c479a-5489-41fe-b2e4-28f033f0e021/init-textfile/0.log"
Nov 01 07:31:38.311821 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:38.311783    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6kh78_a13ea0c7-7fe8-4aec-805c-26197f386653/kube-rbac-proxy-main/0.log"
Nov 01 07:31:38.728934 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:38.728906    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6kh78_a13ea0c7-7fe8-4aec-805c-26197f386653/kube-rbac-proxy-self/0.log"
Nov 01 07:31:39.151257 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:39.151180    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6kh78_a13ea0c7-7fe8-4aec-805c-26197f386653/openshift-state-metrics/0.log"
Nov 01 07:31:40.973725 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:40.973692    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l458f_7e89c29e-4c5c-4b3b-91bc-33014e337036/kuadrant-console-plugin/0.log"
Nov 01 07:31:41.398089 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:41.398009    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f4wws_def3f09f-380f-4342-9f91-740df732a7fe/limitador/0.log"
Nov 01 07:31:42.522178 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:42.522150    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xvv2h_6b81cb01-44c1-4d62-8f59-443a7f27be7d/prometheus-operator/0.log"
Nov 01 07:31:42.936150 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:42.936079    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xvv2h_6b81cb01-44c1-4d62-8f59-443a7f27be7d/kube-rbac-proxy/0.log"
Nov 01 07:31:43.782289 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:43.782253    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d6d64875b-xlxz9_1f330972-954c-4a1b-81e6-223f55770709/telemeter-client/0.log"
Nov 01 07:31:44.199666 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:44.199598    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d6d64875b-xlxz9_1f330972-954c-4a1b-81e6-223f55770709/reload/0.log"
Nov 01 07:31:44.616984 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:44.616958    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d6d64875b-xlxz9_1f330972-954c-4a1b-81e6-223f55770709/kube-rbac-proxy/0.log"
Nov 01 07:31:45.044087 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:45.044061    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584b5d845f-c29z8_09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/thanos-query/0.log"
Nov 01 07:31:45.472546 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:45.472516    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584b5d845f-c29z8_09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/kube-rbac-proxy-web/0.log"
Nov 01 07:31:45.890427 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:45.890353    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584b5d845f-c29z8_09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/kube-rbac-proxy/0.log"
Nov 01 07:31:46.307299 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:46.307273    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584b5d845f-c29z8_09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/prom-label-proxy/0.log"
Nov 01 07:31:46.724435 ip-10-0-138-190 kubenswrapper[5510]: I1101 07:31:46.724407    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584b5d845f-c29z8_09e9bca5-9a75-4cbf-a3b8-7b64ad4f41f8/kube-rbac-proxy-rules/0.log"